Huge speed difference between Tailscale IP and Subnet forwarded real IP

tearfulDalvik commented 1 year ago

What is the issue?

172.16.96.40 and 100.80.80.83 are the same machine. One is the real IP forwarded by tailscale subnet router, one is tailscale interface IP There's huge difference between these IP.

➜  ~ iperf3 -c 172.16.96.40 -R
Connecting to host 172.16.96.40, port 5201
Reverse mode, remote host 172.16.96.40 is sending
[  5] local 100.91.157.44 port 49228 connected to 172.16.96.40 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   431 KBytes  3.53 Mbits/sec
[  5]   1.00-2.00   sec   239 KBytes  1.96 Mbits/sec
[  5]   2.00-3.00   sec   274 KBytes  2.25 Mbits/sec
[  5]   3.00-4.00   sec   225 KBytes  1.84 Mbits/sec
[  5]   4.00-5.00   sec   198 KBytes  1.62 Mbits/sec
[  5]   5.00-6.00   sec   202 KBytes  1.65 Mbits/sec
[  5]   6.00-7.00   sec   190 KBytes  1.56 Mbits/sec
[  5]   7.00-8.00   sec   160 KBytes  1.32 Mbits/sec
[  5]   8.00-9.00   sec   217 KBytes  1.77 Mbits/sec
[  5]   9.00-10.00  sec   171 KBytes  1.40 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  3.62 MBytes  3.04 Mbits/sec                  sender
[  5]   0.00-10.00  sec  2.25 MBytes  1.89 Mbits/sec                  receiver

iperf Done.
➜  ~ iperf3 -c 100.80.80.83 -R
Connecting to host 100.80.80.83, port 5201
Reverse mode, remote host 100.80.80.83 is sending
[  5] local 100.91.157.44 port 49230 connected to 100.80.80.83 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.48 MBytes  12.4 Mbits/sec
[  5]   1.00-2.00   sec  1.87 MBytes  15.7 Mbits/sec
[  5]   2.00-3.00   sec  1.92 MBytes  16.0 Mbits/sec
[  5]   3.00-4.00   sec  1.53 MBytes  12.8 Mbits/sec
[  5]   4.00-5.00   sec  2.45 MBytes  20.7 Mbits/sec
[  5]   5.00-6.00   sec  2.63 MBytes  22.1 Mbits/sec
[  5]   6.00-7.00   sec  1.25 MBytes  10.4 Mbits/sec
[  5]   7.00-8.00   sec  2.04 MBytes  17.1 Mbits/sec
[  5]   8.00-9.00   sec  2.07 MBytes  17.4 Mbits/sec
[  5]   9.00-10.00  sec  1.13 MBytes  9.53 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  18.6 MBytes  15.6 Mbits/sec                  sender
[  5]   0.00-10.00  sec  18.4 MBytes  15.4 Mbits/sec                  receiver

iperf Done

Steps to reproduce

Install tailscale on two machines, and they are not in the same LAN.
Do iPerf3

Are there any recent changes that introduced the issue?

No response

OS

macOS, Windows

OS version

Windows 11, macOS 13.0.1

Tailscale version

1.32.3

Bug report

BUG-820e541cdcdb1fbbdd8a5b9d6ab9ec80de05a91b05544b0556292f718c105280-20221129233039Z-6371398a9c377cee

bradfitz commented 1 year ago

Dup of #414.

DentonGentry commented 1 year ago

I suspect you are going through DERP relays. tailscale ping 100.80.80.83 would say whether a direct or relayed connection is used.

tearfulDalvik commented 1 year ago

Dup of #414.

@bradfitz This is not duplicated with #414 The CPU usage is pretty normal. It is about two different IP(subnet router forwarded real IP and tailnet IP) addresses pointing to the same machine resulting in different speeds.

tearfulDalvik commented 1 year ago

I suspect you are going through DERP relays. tailscale ping 100.80.80.83 would say whether a direct or relayed connection is used.

Yes. I believe both connections are routed through DERP relays

DentonGentry commented 1 year ago

Yes. I believe both connections are routed through DERP relays

That is the thing to fix: DERP rate limits connections being relayed. It looks like both ends of the connection are Hard NAT, and no direct path can be found.

Resolving this could involve:

if one side of the connection is behind a residential gateway like a consumer router, enabling portmapping (UPnP, NAT-PMP, or PCP) can enable it to find a direct connection.
if one side of the connection is in a cloud provider, setting the security group or equivalent to allow UDP port 41641 to ingress can enable it to find a direct connection.

tearfulDalvik commented 1 year ago

Yes. I believe both connections are routed through DERP relays

That is the thing to fix: DERP rate limits connections being relayed. It looks like both ends of the connection are Hard NAT, and no direct path can be found.

Resolving this could involve:

if one side of the connection is behind a residential gateway like a consumer router, enabling portmapping (UPnP, NAT-PMP, or PCP) can enable it to find a direct connection.

if one side of the connection is in a cloud provider, setting the security group or equivalent to allow UDP port 41641 to ingress can enable it to find a direct connection.

Thank you very much for your reply.

I tried the tailnet IP with DERP relay and results in ~15Mbps up/down, which is pretty decent.
However, when I use real IP forwarded by subnet router via DERP relay, I get only like 1Mbps down.
tailnet IP and real IP are the same machine

I think it is more like a software issue instead of derp speed limit.

DentonGentry commented 1 year ago

I think it is worth loading 1.35.30 or later from https://pkgs.tailscale.com/unstable/ onto the subnet router to check throughput. https://tailscale.com/blog/throughput-improvements/

DentonGentry commented 1 year ago

Tailscale 1.36 included the performance improvements described in https://tailscale.com/blog/throughput-improvements/

DentonGentry commented 1 year ago

The 1.40 release contained additional throughput improvements described in Surpassing 10Gb/s over Tailscale

DentonGentry commented 1 year ago

The initial bug report mentions macOS and Windows, and that the subnet router is macOS. This means the subnet routing is being handled in userspace by netstack.

Tailscale releases prior to 1.32.1 had relatively low throughput until 1.32.1 when we fixed an issue with SACK.

aur3l14no commented 1 year ago

I'm having a similar but much milder issue.

Windows running subnet router; 10.xx.xx.xx / 100.xx.xx.xx; iperf3 server
macOS running tailscale / tailscaled; iperf3 client

All running tailscale 1.42.0

The speed of 10.xx.xx.xx (real ip) is about 40% of 100.xx.xx.xx (tailscale ip). CPU usage stays low the whole time.

tailscale / tailscale