Issue with unstable throughput

[ByteDou]

Hi! We downloaded boringtun and ran performance tests, and we found that throughput was inconsistent. Here's a case. for server:

# wireguard-go conf
root@n198-100-016:~# cat /etc/wireguard/wgx.conf
[Interface]
Address = 192.169.1.1/24
ListenPort = 5001
PrivateKey = oHtzn34+s2LB9o8bIFNrvUBTZ7VoXOOc5nXTZOs5jnk=

[Peer]
AllowedIPs = 192.169.1.3/32,192.168.1.3/32
PublicKey = FoGUDBjfh0wu73Ohii3hTVn0NrRxTN4hKeIOVhaaWVQ=
Endpoint = 10.198.100.17:5001
# wireguard-go cmd
wg-quick up wgx

# boringtun conf
root@n198-100-016:~# cat /etc/wireguard/wgu.conf
[Interface]
Address = 192.168.1.1/24
ListenPort = 5000
PrivateKey = oHtzn34+s2LB9o8bIFNrvUBTZ7VoXOOc5nXTZOs5jnk=

[Peer]
AllowedIPs = 192.168.1.3/32
PublicKey = FoGUDBjfh0wu73Ohii3hTVn0NrRxTN4hKeIOVhaaWVQ=
Endpoint = 10.198.100.17:5000

# boringtun cmd
su zhaoya
sudo WG_QUICK_USERSPACE_IMPLEMENTATION=boringtun wg-quick up wgu

iperf -s 

for client:

# wireguard-go conf
root@n198-100-017:~# cat /etc/wireguard/wgx.conf
[Interface]
Address = 192.169.1.3/24
ListenPort = 5001
PrivateKey = sFWijh452FMN5krVhsF1wOoyK8A2aiEg8cUXJuWOX08=

[Peer]
AllowedIPs = 192.169.1.1/32,192.168.1.1/32
PublicKey = 4+b3CbE4qmzJbheQTm4pCC9Ksx8pnHbncGAmgC/oDjY=
Endpoint = 10.198.100.16:5001
PersistentKeepalive = 25

# wireguard-go cmd 
wg-quick up wgx

# boringtun conf
root@n198-100-017:~# cat /etc/wireguard/wgu.conf
[Interface]
Address = 192.168.1.3/24
ListenPort = 5000
PrivateKey = sFWijh452FMN5krVhsF1wOoyK8A2aiEg8cUXJuWOX08=

[Peer]
AllowedIPs = 192.168.1.1/32
PublicKey = 4+b3CbE4qmzJbheQTm4pCC9Ksx8pnHbncGAmgC/oDjY=
Endpoint = 10.198.100.16:5000

# boringtun cmd
su zhaoya
sudo WG_QUICK_USERSPACE_IMPLEMENTATION=boringtun wg-quick up wgu

iperf -c 10.198.100.16 -i 1

for result:

We changed the architecture and created multiple reuseport sockets for each thread. Each thread operated only on its own socket and the performance was very stable. Here's our patch: multi_poll.diff.txt

I don't know if this issue is known, and if the community has a fix or optimization plan.

Looking forward to reply.

[Noah-Kennedy]

I'm currently in the middle of a rewrite which should improve things, I just haven't had the bandwidth to finish this yet.

[ByteDou]

I'm currently in the middle of a rewrite which should improve things, I just haven't had the bandwidth to finish this yet.

Thanks for your reply.

Have you encountered the bandwidth issue I mentioned?

In addition, as you said, you are rewriting boringtun and could you tell me some details about your work? We are also working on some optimizations, so we don't want to reinvent the wheel.

[Noah-Kennedy]

@ByteDou I'm getting rid of the custom epoll loop that we have and moving this over to tokio. This should also make it a bit easier to just shard this out and use reuseport.