Are direct connections working?

[orrc]

Thanks for this guide!

However, I'm not sure what I'm doing wrong here, but I don't seem to be able to get a direct connections to the nodes on Fly. i.e. tailscale status and tailscale ping shows that connections are going through a DERP relay:

$ /Applications/Tailscale.app/Contents/MacOS/Tailscale status
…
100.125.56.76   fly-fra      chris@       linux   active; exit node; relay "fra", tx 3185 rx 5303

$ /Applications/Tailscale.app/Contents/MacOS/Tailscale ping fly-fra
pong from fly-fra (100.125.56.76) via DERP(fra) in 65ms
pong from fly-fra (100.125.56.76) via DERP(fra) in 57ms
…
2022/03/07 22:09:24 direct connection not established

The same happened when launching the Fly app in fra, ams, or lhr.

Which seems unexpected, as the Fly config asks for UDP port 41641 to be open, which is normally all that's needed to establish a direct connection. For example, on another exit node I have:

$ /Applications/Tailscale.app/Contents/MacOS/Tailscale status
…
100.78.27.79    xyz          chris@       linux   active; offers exit node; direct 89.14.247.123:41641, tx 9001 rx 9002

Are you folks using this setup seeing Tailscale able to make direct connections to the Fly app? Am I missing something?

[bslatyer]

Hey @orrc,

I've got two nodes (LAX & SYD) and they also go via DERP. I don't think it's a big issue that it I going via DERP as long as you can connect to them as Exit Nodes.

Here is a little more about DERP on the Tailscale website

Some especially cruel networks block UDP entirely, or are otherwise so strict that they simply cannot be traversed using STUN and ICE.

For those situations, Tailscale provides a network of so-called DERP (Designated Encrypted Relay for Packets) servers. These fill the same role as TURN servers in the ICE standard, except they use HTTPS streams and WireGuard keys instead of the obsolete TURN recommendations.

[orrc]

Hey @bslatyer, thanks for the confirmation that other folk are also unable to make a direct connection.

You're right that I can successfully use the Fly app as an exit node, but DERP relays ideally shouldn't be used full-time, as they're primarily intended for cases where a direct connection can't be negotiated at all, e.g. "especially cruel networks [that] block UDP entirely".

Since the Fly app gets a public IP address, and the app configuration exposes the correct UDP port, this should be the ideal scenario for Tailscale to establish a direct connection. So I should really be able to get the benefits of improved latency, bandwidth, and not unnecessarily burdening Tailscale's infrastructure.

---

However, comparing the public IP addresses allocated to the Fly app (fly ips list) to those reported by Tailscale running in the Fly app (tailscale status -json | jq -r '.Peer | objects[] | select(.HostName == "fly-lhr").CurAddr'), I notice that they are completely different.

Thinking about it some more, I guess the way that Fly works means it won't be possible to connect directly. The public IPs allocated by Fly are anycast addresses, whereas the public IP that Tailscale running in the Fly app sees (and reports to its peers) will be in a block specific to the region it's running in — probably behind a NAT, and not directly accessible.

So unless my laptop that's trying to use the exit node has port 41641 publicly exposed (which is unlikely in almost all cases), then I imagine direct connections won't be possible when running Tailscale on Fly…

Thanks for the chance to do some rubber-duck debugging.

[CMCDragonkai]

@orrc I noticed that the dedicated anycast IPv4 that is assigned to the fly app using flyctl ips allocate-v4 is not the same IP that is being used by tailscale during a direct connection, and it's also not the IP that shows up when you google "whats my ip" after getting the exit node routing working.

I wonder, what exactly is the difference between the direct IP that tailscale ends up using, and the IP that is being allocated to the app.

And if we aren't even using the IP being allocated to the app, what's the point of allocating a dedicated anycast IP to the app at all?

Furthermore is there a way to actually find that IP in fly's dashboard? It doesn't seem to show that IP that is actually being used.

[CMCDragonkai]

Thinking about it some more, I guess the way that Fly works means it won't be possible to connect directly. The public IPs allocated by Fly are anycast addresses, whereas the public IP that Tailscale running in the Fly app sees (and reports to its peers) will be in a block specific to the region it's running in — probably behind a NAT, and not directly accessible.

Also this was not true for me, I was able to ping that direct IP, and tailscale status shows that I'm connecting to fly-syd directly.