Packets block happens on writing to wireguard device by udp protocol exceeds 30 packets/sec

bigboyq commented 2 years ago

Package version

1.0.20211208-1

Firmware version

v2.0.9-hotfix.2

Device

EdgeRouter X (SFP) - e50

Issue description

My network connection looks like ''' Server[192.168.100.18]------Wireguard-------ERX[192.168.100.19/192.168.0.1]-------LAN-------Mac[192.168.0.110] ''' 192.168.100.18/192.168.100.19 is wireguard devices 192.168.0.1/192.168.0.110 is Lan devices if I run on Server

iperf3 -s

run on Mac

iperf3 -u -c 192.168.100.18 -l 99

I got log on Server

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.0.110, port 50429
[ 5] local 192.168.100.18 port 5201 connected to 192.168.0.110 port 58909
[ ID] Interval      Transfer   Bitrate     Jitter  Lost/Total Datagrams
[ 5]  0.00-1.00  sec 2.90 KBytes 23.7 Kbits/sec 15644245684.792 ms 0/30 (0%)  
[ 5]  1.00-2.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  2.00-3.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  3.00-4.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  4.00-5.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  5.00-6.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  6.00-7.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  7.00-8.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  8.00-9.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval      Transfer   Bitrate     Jitter  Lost/Total Datagrams
[ 5]  0.00-10.01 sec 2.90 KBytes 2.37 Kbits/sec 15644245684.792 ms 0/30 (0%) receiver

Obviously, while udp packets writing to wireguard device exceeds 30p/s, the device blocked writing. I think something might happens on packet queue located inside wireguard driver.

Other tests been done

If sending from Server to Mac by udp over wireguard , No blocking
If NAT enable/disable on ERX, No difference
If iperf3 using TCP, No blocking

Configuration and log output

No response

bigboyq commented 2 years ago

More tests has been done

If run iperf3 -c directly on Er-x, no packet loss
If tune the -l from 99 to 64, Server got 0/103, then 0/0
if tune the -l from 99 to 32, Server got 0/183, then 0/0 If change server from ping 9ms to 45ms, test results keep the same

yukinobu commented 1 year ago

Hi.

I'm also using EdgeRouter X and faced a similar issue. From a client computer which connected to EdgeRouter's WireGuard server, I could browse web sites such as YouTube, but the video playback is very slow.

When I inspect the packet flow with Wireshark, it seems that YouTube first sends me the video data over UDP. But EdgeRouter dropped these UDP packets. After I waited a while it seemed to fall back to TCP and and I was able to play the video.

Note: the link above points to the hwnat issue. When I disabled my EdgeRouter's hwnat, the issue no longer occurred. However, the EdgeRouter's performance will drop significantly...

jbeez commented 1 year ago

@yukinobu Was it UDP over port 443(aka QUIC)? You might be able to just block QUIC outright so it never uses that and see if it improves the experience?

yukinobu commented 1 year ago

@jbeez Yes, it seemed QUIC. Now, I blocking all egress UDP packets and it improves my experience definitely!

Below is a snippet of my NAT rules. For packets from WireGuard tunnels, only TCP and ICMP are handled by NAT.

rule 5030 {
    description "masquerade from WireGuard to WAN (exclude udp to avoid hwnat bug)"
    log disable
    outbound-interface eth0
    protocol tcp
    source {
        address 10.xxx.yyy.0/24
    }
    type masquerade
}
rule 5031 {
    description "masquerade from WireGuard to WAN (exclude udp to avoid hwnat bug)"
    log disable
    outbound-interface eth0
    protocol icmp
    source {
        address 10.xxx.yyy.0/24
    }
    type masquerade
}

WireGuard / wireguard-vyatta-ubnt