Improving Wireguard upload speed

[ppittle]

This is a general question and unrelated to the recent v055x upload issues (https://github.com/celzero/rethink-app/issues/1306)

Is there a guide to improve Wireguard tunnel upload speed?

If I connect to my home network via the official Wireguard client and run a speedtest.net speed test, the upload bandwidth is much better (10x) than running the same wg config through Rethink DNS's Wireguard tunnel. Upload on Rethink seems to always cap out around 13 Mbps.

Android Client	Down	Up
Wireguard 1.0.20231018	68.04 Mbps	131.81 Mbps
Rethink DNS v0.5.5i	70.15 Mbps	7.84 Mbps

Test Site: www.speedtest.net Browser: Brave for Android OS: Android 14 - GrapheneOS

[ignoramous]

We've noticed this too, but it is unclear why the upload speeds are slow in our implementation. We'll keep looking and update here if we manage to root-cause it.

If you're technical enough, you can eval the code here:

upstream WireGuard (before gso/gro): https://github.com/WireGuard/wireguard-go/blob/ebbd4a433088a06277b7e7bc6e13322c167d5554/conn/bind_std.go#L166-L191

rethink wireguard (without gso/gro): https://github.com/celzero/firestack/blob/d92f39862240c5686e94036af3ad5fe65979bae1/intra/ipn/wg/wgconn.go#L265-L311

[Lanius-collaris]

@ignoramous Every time rethink calls Send(), log.V() and loge() are called. If I remove them and loge() in makeReceiveFn(), I can get 2.19 Gbits/sec in the emulator (My CPU is i3-12100, with logging I can only get 156 Mbits/sec)

... LAN traffic leaked, so I got incorrect results. The iperf3 server and the wireguard peer are on host, even the iperf client can connect peer ip without rethink.🙃

[ppittle]

I'm guessing log.V() is a verbose log call, and from my limited understanding of Go, appears to be in the critical path.

Can the logging calls be dropped? At least for release builds?

[ignoramous]

Thanks a tonne for your analysis.

---

log.V shouldn't do anything heavy if the app isn't setup to output Verbose or Very Verbose in Configure -> Settings -> Log level. I'm surprised removing those lines is as effective.

I'll run some tests the first thing tomorrow and get back to you!

You can also opt to send a pull request (no pressure) on https://github.com/celzero/firestack by guarding log.V() (but not log.E) invocations in makeReceiveFunc, send, and loge with a settings.Debug:

 import (
..
+ github.com/celzero/firestack/intra/settings
..
 )

..

func makeReceiveFn(...
..
+ if settings.Debug {
+     log.V(...
+ }
..

[Lanius-collaris]

Sorry. Yesterday I imported same .conf file into wireguard-android and rethink.

[Interface]
Address = 10.64.0.6/24, 2400:1111::6/64
MTU = 1400
PrivateKey = key
DNS = 9.9.9.9

[Peer]
AllowedIPs = 10.64.0.5/32
#iperf3 server's IP is 10.64.0.5
Endpoint = 192.168.1.72:10003
PublicKey = key

That is why traffic leaks. 🤦Today I retested with AllowedIPs = 0.0.0.0/0, ::/0, Upload seems around 200 Mbps. I'm surprised that rethink firewall mode without wireguard is much slower than wireguard.

setup Create a virtual device in Android Studio Start wireguard on Host Run iperf3 -s on Host

iperf3 -c 10.64.0.5	Client	Up
Wireguard 1.0.20231018	1.2 Gbps
Rethink DNS v0.5.5d firewall mode	317 Mbps
Rethink DNS v0.5.5d DNS+firewall with wireguard	206 Mbps

[ignoramous]

I'm surprised that rethink firewall mode without wireguard is much slower than wireguard.

Sorry, doesn't the table show 317mbps for firewall vs 206mbps for firewall+wg

That is why traffic leaks.

Was that the correct behaviour? You mean, traffic shouldn't leak?

[Lanius-collaris]

I'm surprised that rethink firewall mode without wireguard is much slower than wireguard.

Sorry, doesn't the table show 317mbps for firewall vs 206mbps for firewall+wg

🤦 Sorry. The second "wireguard" should be com.wireguard.android.

That is why traffic leaks.

Was that the correct behaviour? You mean, traffic shouldn't leak?

🤔 2.19 Gbits/sec is very unusual, so I think traffic was not passing through TUN device, maybe "Do not route Private IPs" was enable? However I can't reproduce with that .conf file now.

[Lanius-collaris]

Two snapshots from FlightRecorder Captured while running iperf3 -c 10.64.0.5 -t 1 firestack-trace.zip

[Lanius-collaris]

@ignoramous @ppittle Please try this patch and set Error level in Configure -> Settings -> Log level. Seems that it doubles the upload speed for firewall mode.

[ignoramous]

Thanks. We've identified a string of other minor fixes that has bumped up the upload speed for WireGuard proxies (and in our prelim tests, appear on-par with download speeds; though, nothing spectacular).

The fix will land in v055k (the next version).

[Lanius-collaris]

😱 Does it regress? I remember that 760a9a47 can archieve 600-700 Mbps (upload) on my device (emulator), however v055l can only archieve about 200 Mbps, seems as slow as v055i.

[ignoramous]

seems as slow as v055i

The current patchset has improved WireGuard upload speeds. Unsure about the overall upload speeds.

Does it regress? I remember that 760a9a47

May be. But 760a9a47 was prone to deadlocks and so we had to make more changes, (though, those changes don't strike me as degrading anything).

[Lanius-collaris]

@ignoramous Thanks. I'm reassured. Emulator's bottlenecks may be different.

🤦I forgot most physical phones don't have 1G wireguard connection.

[ppittle]

Thanks. We've identified a string of other minor fixes that has bumped up the upload speed for WireGuard proxies (and in our prelim tests, appear on-par with download speeds; though, nothing spectacular).

The fix will land in v055k (the next version).

Excited for this to hit f-droid!

[ppittle]

Bad news I'm afraid, retried with v0.5.5l and upload perf is still not close to native WireGuard client performance.

If I manually set MTU to 1412 on my connection, I could get it up to nearly 20mbps, but that's still a big gap of the 100 mpbs I see with WireGuard:

Android Client	Down	Up
Wireguard 1.0.20231018	162.85 Mbps	102.56 Mbps
Rethink DNS v0.5.5l	78.19 Mbps	19.94 Mbps

NOTE: I followed @Lanius-collaris advice and set Error level in Configure -> Settings -> Log level.

Is there anything else I should check??

[ignoramous]

(the official wireguard is about get much faster, but rethink will always be limited due to how it processes egress packets in userspace to apply firewall rules and what-not)

The fixes to speed up uploads are in v055m but it doesn't work with

• Peer endpoints with DNS name (as opposed to IP addr) #1490 (cause yet unknown)
• Peer endpoints with IPv6 addrs (but on IPv4 networks) #1483 (since fixed)

In our tests, we hit 45mbps down and 50mbps up at fiber.google.com/speedtest

[ppittle]

Just tried v055m huge improvement!

Android Client	Down	Up
Wireguard 1.0.20231018	101.85Mbps	97.75 Mbps
Rethink DNS v0.5.5m	110.13Mbps	91.94 Mbps

If it helps with #1490 - I'm connecting to my home network (OpnSense) using a DNS name.

Thanks for all the work on this! This was the last big blocker preventing adopting Rethink as a daily driver.

[ignoramous]

This was the last big blocker

Interesting. Note though, Rethink doesn't run WireGuard at L3 (IP) but instead at L4 (TCP, UDP).

Thanks for all the work on this!

Thanks for the tests and this bug report (which prompted @Lanius-collaris to delve further and eventually lead us to a fix).

[ppittle]

@ignoramous - apologies for turning this into a discussion, but do you have any documentation on this statement:

Rethink doesn't run WireGuard at L3 (IP) but instead at L4 (TCP, UDP)

Specifically, I don't fully grok the implications for privacy / security / performance. Could you elaborate?

[ignoramous]

Specifically, I don't fully grok the implications for privacy / security / performance.

Rethink doesn't hand over the TUN device to WireGuard. And so, WireGuard only sees what Rethink sends it over an "emulated" a TUN device. Rethink only sends TCP, ICMP, and UDP packets to WireGuard over that emulated device, drops all others (no L3 protos like ARP, IGMP etc, or even other L4 protos like RTP, SCTP). As a special case, Rethink redirects ICMP (L3) and DNS (L7 over TCP or UDP) to underlying network (mobile/wifi/usb/ethernet) when running WireGuard in Advanced mode.

As for privacy/security, there's is no change, as it is all still WireGuard.

As for performance, the emulation has a cost, though we can improve still. For Rethink, which is specifically meant to run on Androids, 100mbps is sufficient (unlike for WireGuard run on servers or in large enterprises).

Running WireGuard at L4 is not set in stone. Rethink can run it at L3, but it would either need root or undergo an overhaul of its network engine (also doable).