search

processone / eturnal

STUN / TURN standalone server
https://eturnal.net
Apache License 2.0
237 stars 22 forks source link

Proxy Protocol - eturnal behind reverse proxy #18

Closed sando38 closed 2 years ago

sando38 commented 2 years ago

Dear Holger,

I would like to load balance eturnal(s) with an nginx load balancer for example. turn obviously needs the real IP-address of a client asking for relaying. ejabberd has implemented proxy-protocol, at least for its xmpp listeners. So I wonder if this is planned also for eturnal. A possible nginx config could look like this:

stream {
    upstream traefik {
        server traefik-lb:443;
    }

    upstream turnserver {
        server eturnal1:5349;
        server eturnal2:5349;
        server eturnal3:5349;
    }

    map $ssl_preread_alpn_protocols $upstream {
        default traefik;
        "stun.turn" turnserver;
        "stun.nat-discovery" turnserver;
    }

    server {
        listen 443;

        ssl_preread on;
        proxy_pass $upstream;
        proxy_protocol on;
    }
}

Thanks in advance! Have a good evening, Sando

weiss commented 2 years ago

The TURN protocol works in an asymmetric manner. There's three parties involved: The client, the server, and the peer (actually, there can be multiple peers, but that's not commonly used in practice). So there's two connections:

CLIENT <-- (1) -- [port 3478] -> SERVER <- [random high port] -- (2) --> PEER

Connection (1) can use UDP, TCP, or TLS transports. Connection (2) is always UDP. It works like this: The client connects to the server via UDP, TCP, or TLS on a known port (conventionally 3478 or 5349), authenticates, and asks the server to allocate a dedicated UDP port for the peer to talk to. This will be a random port within the relay_min_port to relay_max_port range, chosen by the server (say, 51234) and communicated back to the client, which then communicates it to the peer. Any traffic sent by the peer to that port over connection (2) will be relayed to the client over connection (1), and vice versa.

I don't see how proxying such connections could work without the proxy understanding the TURN protocol and performing deep packet inspection on connection (1). I don't think this is possible with HAproxy and I'm sure it isn't with Nginx. (And if it was, I'm not sure I'd see much point in having the proxy forward the traffic to the TURN server rather than just performing the relaying itself.)

sando38 commented 2 years ago

Sorry Holger, I meant STUN for discovering IP addresses. Nevertheless, thanks for the answer. And, yes, you are right with your explanation with the abilities of HAProxy and Nginx with UDP and proxy forwarding. If your answer is applicable as well to STUN, please just let me know and close the "issue". Thanks and have a great evening!

weiss commented 2 years ago

Ah, no, my response doesn't apply to plain STUN (at least not via TCP), except for the "don't quite see the point" part: STUN queries/responses are tiny and trivial, not sure why you'd bother having a proxy forward those packets rather than just having a STUN server responding directly?

Then again, seems Coturn does support this, with the reasoning that

some network setups will require using a TCP reverse proxy in front of the STUN server.

sando38 commented 2 years ago

Thank you for confirming my thoughts regarding STUN (for the TCP/TLS part). I am happy to describe you my simplified use case, which is however nothing new and probably also the reason why Coturn implemented it as well.

You are right, load balancing STUNs isn't the topic, the loadbalancing aspect in my initial post was more related to the complete package "eturnal" STUN/ TURN :)

Thank you and wishing you a nice evening.

weiss commented 2 years ago

Ah! I don't think supporting the proxy protocol actually helps clients behind restrictive firewalls, because they'd use STUN for establishing a p2p connection which such a firewall would then block. So they'll need TURN anyway, which doesn't require the client's IP address and will therefore work without proxy protocol.

However, I see how this makes sense for the case where you prefer to offer STUN/TURN only via proxy, i.e. where STUN clients that are not behind restrictive firewalls will use port 443 as well.

I'll leave this issue open and think about implementing the feature.

weiss commented 2 years ago

Could you easily test a patch? (I could build DEB/RPM/binary packages if that helps.)

sando38 commented 2 years ago

Cool! Yes, I gladly assist testing. I am able to test from tomorrow evening on.

P.S. yes, binary packages would be great. P.P.S I could also test with a Docker-Image, if that makes things easier.

weiss commented 2 years ago

I gladly assist testing. I am able to test from tomorrow evening on.

Perfect! The packages currently available on https://eturnal.net/tmp/ should accept a listen configuration such as the following:

listen:
  -
      ip: "::"
      port: 3478
      transport: tcp # or 'auto' to accept TLS connections as well.
      proxy_protocol: true
    -
      ip: "::"
      port: 5349
      transport: tls # or 'auto' to accept unencrypted TCP connections as well.
      proxy_protocol: true

If things work as expected, you should see [info] level log lines like this one:

Accepting proxied connection: 198.51.100.42:50111 -> 203.0.113.2:3478

… where 198.51.100.42 is the real client address.

If you're using XMPP, you could join our room (eturnal@conference.process-one.net), otherwise feel free to email me in case anything is unclear or doesn't work.

weiss commented 2 years ago

I found some time to test this myself today, seems to work.

Typical STUN/TURN clients don't support ALPN though, so the following part of your Nginx configuration example will probably not work as expected (I'm aware of the example configuration in the XMPP wiki which should probably be changed):

map $ssl_preread_alpn_protocols $upstream {
    default traefik;
    "stun.turn" turnserver;
    "stun.nat-discovery" turnserver;
}

One solution might be using a separate host name for STUN/TURN and then routing the traffic based on SNI (which modern clients do seem to support) instead.

sando38 commented 2 years ago

Dear Holger, yes, also finished my testing... and also discovered that ALPN types are not really supported for STUN/TURN, at least not for ConversationsIM, SiskinIM and BeagleIM. From your point of view: Are those two ALPN types "depreciated" or not really useful, so that client developers so far did decide against implementing it into the "clienthello"?

Nevertheless, thanks for implementation and your efforts. Wishing you a great Sunday.

weiss commented 2 years ago

Are those two ALPN types "depreciated" or not really useful, so that client developers so far did decide against implementing it into the "clienthello"?

ALPN is a relatively new TLS extension and the STUN/TURN libraries used by those clients (such as Google's libwebrtc) obviously weren't updated to support it yet. I guess the demand among WebRTC service providers just hasn't been strong enough so far (they often use dedicated TURN servers).

sando38 commented 2 years ago

Yes, I agree with the argument as it makes sense to have TURN servers on dedicated machines in most cases.

For completion of testing, I provide you my test logs for TURN relay, as it throws two errors, however functions correctly:

STUN/ TURN - TURN Test

2021-12-12 17:28:11.720677+01:00 [info] Listening on xxx.xxx.xxx.xxx(host-public/internal-IP):3478 (udp) (STUN/TURN) (eturnal:start_listeners/0:297)
2021-12-12 17:28:11.721309+01:00 [info] Listening on xxx.xxx.xxx.xxx(host-public/internal-IP):5349 (auto) (STUN/TURN) (eturnal:start_listeners/0:297)
2021-12-12 17:28:11.721340+01:00 [info] Applied new listen configuration (eturnal:apply_config_changes/2:481)
2021-12-12 17:28:47.347413+01:00 [info] Accepting proxied connection: xxx.xxx.xxx.xxx(client-IP):60246 -> xxx.xxx.xxx.xxx(nginx-lb-IP):8443 [TCP|TLS, session b6] (stun_listener:accept/2:191)
2021-12-12 17:28:47.350951+01:00 [error] Ignoring unknown option '{sock_peer_name,
                             {{{xxx,xxx,xxx,xxx(nginx-lb-IP)},8443},
                              {{xxx,xxx,xxx,xxx(client-IP)},60246}}}' [TLS, session b6, anonymous, client xxx.xxx.xxx.xxx(client-IP):60246] (stun:prepare_state/4:579)
2021-12-12 17:28:47.351058+01:00 [error] Ignoring unknown option 'proxy_protocol' [TLS, session b6, anonymous, client xxx.xxx.xxx.xxx(client-IP):60246] (stun:prepare_state/4:579)
2021-12-12 17:28:47.419795+01:00 [info] Accepting long-term STUN/TURN authentication [TLS, session b6, user 1639369724:47ea473ceaaa269e, client xxx.xxx.xxx.xxx(client-IP):60246] (stun:process/2:249)
2021-12-12 17:28:47.421323+01:00 [notice] Creating TURN allocation (lifetime: 599 seconds) [TLS, session b6, user 1639369724:47ea473ceaaa269e, client xxx.xxx.xxx.xxx(client-IP):60246, relay xxx.xxx.xxx.xxx(host-public/internal-IP):60042] (turn:wait_for_allocate/2:205)
2021-12-12 17:28:49.620583+01:00 [info] Accepting proxied connection: xxx.xxx.xxx.xxx(client-IP):21243 -> xxx.xxx.xxx.xxx(nginx-lb-IP):8443 [TCP|TLS, session bm] (stun_listener:accept/2:191)
2021-12-12 17:28:49.621788+01:00 [error] Ignoring unknown option '{sock_peer_name,
                             {{{xxx,xxx,xxx,xxx(nginx-lb-IP)},8443},
                              {{xxx,xxx,xxx,xxx(client-IP)},21243}}}' [TLS, session bm, anonymous, client xxx.xxx.xxx.xxx(client-IP):21243] (stun:prepare_state/4:579)
2021-12-12 17:28:49.622026+01:00 [error] Ignoring unknown option 'proxy_protocol' [TLS, session bm, anonymous, client xxx.xxx.xxx.xxx(client-IP):21243] (stun:prepare_state/4:579)
2021-12-12 17:28:49.684382+01:00 [info] Accepting long-term STUN/TURN authentication [TLS, session bm, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):21243] (stun:process/2:249)
2021-12-12 17:28:49.684774+01:00 [notice] Creating TURN allocation (lifetime: 600 seconds) [TLS, session bm, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):21243, relay xxx.xxx.xxx.xxx(host-public/internal-IP):55983] (turn:wait_for_allocate/2:205)
2021-12-12 17:28:49.835756+01:00 [info] Accepting long-term STUN/TURN authentication [TLS, session bm, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):21243] (stun:process/2:249)
2021-12-12 17:28:49.835972+01:00 [info] Client requested closing the TURN session [TLS, session bm, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):21243, relay xxx.xxx.xxx.xxx(host-public/internal-IP):55983] (turn:active/2:260)
2021-12-12 17:28:49.836106+01:00 [info] Deleting TURN allocation [TLS, session bm, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):21243, relay xxx.xxx.xxx.xxx(host-public/internal-IP):55983] (turn:terminate/3:452)
2021-12-12 17:28:49.836226+01:00 [info] Connection reset by peer [TLS, session bm, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):21243] (stun:handle_info/3:171)
2021-12-12 17:28:49.836175+01:00 [notice] Relayed 0 KiB (in 0 B / 0 packets, out 0 B / 0 packets), duration: 0 seconds [TLS, session bm, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):21243, relay xxx.xxx.xxx.xxx(host-public/internal-IP):55983] (turn:terminate/3:459)
2021-12-12 17:28:49.924154+01:00 [info] Accepting proxied connection: xxx.xxx.xxx.xxx(client-IP):29977 -> xxx.xxx.xxx.xxx(nginx-lb-IP):8443 [TCP|TLS, session cy] (stun_listener:accept/2:191)
2021-12-12 17:28:49.926409+01:00 [error] Ignoring unknown option '{sock_peer_name,
                             {{{xxx,xxx,xxx,xxx(nginx-lb-IP)},8443},
                              {{xxx,xxx,xxx,xxx(client-IP)},29977}}}' [TLS, session cy, anonymous, client xxx.xxx.xxx.xxx(client-IP):29977] (stun:prepare_state/4:579)
2021-12-12 17:28:49.926497+01:00 [error] Ignoring unknown option 'proxy_protocol' [TLS, session cy, anonymous, client xxx.xxx.xxx.xxx(client-IP):29977] (stun:prepare_state/4:579)
2021-12-12 17:28:49.991264+01:00 [info] Accepting long-term STUN/TURN authentication [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977] (stun:process/2:249)
2021-12-12 17:28:49.991625+01:00 [notice] Creating TURN allocation (lifetime: 600 seconds) [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977, relay xxx.xxx.xxx.xxx(host-public/internal-IP):55592] (turn:wait_for_allocate/2:205)
2021-12-12 17:28:50.023015+01:00 [info] Accepting long-term STUN/TURN authentication [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977] (stun:process/2:249)
2021-12-12 17:28:50.023159+01:00 [info] Creating TURN permission for 10.1.100.105 [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977, relay xxx.xxx.xxx.xxx(host-public/internal-IP):55592] (turn:update_permissions/2:492)
2021-12-12 17:28:52.428053+01:00 [info] Accepting long-term STUN/TURN authentication [TLS, session b6, user 1639369724:47ea473ceaaa269e, client xxx.xxx.xxx.xxx(client-IP):60246] (stun:process/2:249)
2021-12-12 17:28:52.428252+01:00 [info] Connection reset by peer [TLS, session b6, user 1639369724:47ea473ceaaa269e, client xxx.xxx.xxx.xxx(client-IP):60246] (stun:handle_info/3:171)
2021-12-12 17:28:52.428360+01:00 [info] Client requested closing the TURN session [TLS, session b6, user 1639369724:47ea473ceaaa269e, client xxx.xxx.xxx.xxx(client-IP):60246, relay xxx.xxx.xxx.xxx(host-public/internal-IP):60042] (turn:active/2:260)
2021-12-12 17:28:52.428457+01:00 [info] Cannot respond to client: Connection closed [TLS, session b6, user 1639369724:47ea473ceaaa269e, client xxx.xxx.xxx.xxx(client-IP):60246, relay xxx.xxx.xxx.xxx(host-public/internal-IP):60042] (turn:send/2:514)
2021-12-12 17:28:52.428528+01:00 [info] Deleting TURN allocation [TLS, session b6, user 1639369724:47ea473ceaaa269e, client xxx.xxx.xxx.xxx(client-IP):60246, relay xxx.xxx.xxx.xxx(host-public/internal-IP):60042] (turn:terminate/3:452)
2021-12-12 17:28:52.428619+01:00 [notice] Relayed 0 KiB (in 0 B / 0 packets, out 0 B / 0 packets), duration: 5 seconds [TLS, session b6, user 1639369724:47ea473ceaaa269e, client xxx.xxx.xxx.xxx(client-IP):60246, relay xxx.xxx.xxx.xxx(host-public/internal-IP):60042] (turn:terminate/3:459)
2021-12-12 17:28:52.776965+01:00 [info] Accepting long-term STUN/TURN authentication [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977] (stun:process/2:249)
2021-12-12 17:28:52.777114+01:00 [info] Creating TURN permission for xxx.xxx.xxx.xxx(host-public/internal-IP) [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977, relay xxx.xxx.xxx.xxx(host-public/internal-IP):55592] (turn:update_permissions/2:492)
2021-12-12 17:28:52.946502+01:00 [info] Accepting long-term STUN/TURN authentication [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977] (stun:process/2:249)
2021-12-12 17:28:52.946744+01:00 [info] Refreshing TURN permission for xxx.xxx.xxx.xxx(host-public/internal-IP) [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977, relay xxx.xxx.xxx.xxx(host-public/internal-IP):55592] (turn:update_permissions/2:492)
2021-12-12 17:28:52.946812+01:00 [info] Binding TURN channel 4001 for peer xxx.xxx.xxx.xxx(host-public/internal-IP):49612 [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977, relay xxx.xxx.xxx.xxx(host-public/internal-IP):55592] (turn:active/2:342)
2021-12-12 17:28:57.018616+01:00 [info] Accepting long-term STUN/TURN authentication [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977] (stun:process/2:249)
2021-12-12 17:28:57.018778+01:00 [info] Client requested closing the TURN session [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977, relay xxx.xxx.xxx.xxx(host-public/internal-IP):55592] (turn:active/2:260)
2021-12-12 17:28:57.018884+01:00 [info] Deleting TURN allocation [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977, relay xxx.xxx.xxx.xxx(host-public/internal-IP):55592] (turn:terminate/3:452)
2021-12-12 17:28:57.018922+01:00 [notice] Relayed 28 KiB (in 15363 B / 191 packets, out 13358 B / 198 packets), duration: 7 seconds [TLS, session cy, user 1639369726:e7d04d41eaaa269e, client xxx.xxx.xxx.xxx(client-IP):29977, relay xxx.xxx.xxx.xxx(host-public/internal-IP):55592] (turn:terminate/3:459)
weiss commented 2 years ago

that makes sense

Either way, as I said, you should be able to work around missing ALPN support by adding e.g. a CNAME turn.example.com -> example.com, pointing clients to that name in your XMPP server configuration, and then using a map like this one:

map $ssl_preread_server_name $upstream {
    default traefik;
    turn.example.com turnserver;
}

as it throws two errors, however functions correctly

Yes, the bogus errors were fixed in the meantime, I've now uploaded new binaries to https://eturnal.net/tmp in case you're interested. Many thanks for testing this, glad to hear it's working!

licaon-kter commented 2 years ago

One solution might be using a separate host name for STUN/TURN and then routing the traffic based on SNI (which modern clients do seem to support) instead.

Best option so far, yes https://github.com/yrutschle/sslh/issues/246

sando38 commented 2 years ago

Yep, I did the mapping with server-name instead. Just needed to change some things in the configs first, but it works so far. Thanks again for the help! ... and was a pleasure to support with testing.. least thing I could do. :) Wishing both a great evening.