search

johanhelsing / matchbox

Painless peer-to-peer WebRTC networking for rust wasm (and native!)
Apache License 2.0
888 stars 73 forks source link

Setting up Own Server to Host a Matchbox Server #345

Closed PraxTube closed 1 year ago

PraxTube commented 1 year ago

I am using my own server to host a matchbox server and I tried to change it from only working locally to working globally. I tested it with my public IP address and a forwarded port, but it doesn't seem to work. The peers are able to connect to the server (see logs below), but it isn't able to establish a proper connection. Note that I am testing with one peer on the same local network as the server and one peer on a remote network.

When I have both peers in the local network things work fine. They connect without issue and my game plays like it did before. I didn't test to have both peers be remote, but if that is gonna work I am gonna lose it.

In the following 56.135.57.45 is my public IP address. Both ports 2124 and 3536 have been forwarded.

Game Logs

The logs of the different network parties.

Matchbox Server Log

my-server$ matchbox_server 0.0.0.0:2124
2023-10-21T09:47:12.439671Z  INFO Matchbox Signaling Server: 0.0.0.0:2124
2023-10-21T09:47:17.469757Z  INFO Client connected 46.114.176.114:22895: PeerId(30b14bd8-3a85-470d-8dd8-ce9365ce4ac2)
2023-10-21T09:47:18.863468Z  INFO Client connected 56.135.57.45:42516: PeerId(fae1d11e-307c-4cfa-92b3-8054264e724c)
2023-10-21T09:47:18.863806Z  INFO 30b14bd8-3a85-470d-8dd8-ce9365ce4ac2 -> "{\"NewPeer\":\"fae1d11e-307c-4cfa-92b3-8054264e724c\"}"
2023-10-21T09:49:28.452442Z  WARN Unrecoverable error with PeerId(30b14bd8-3a85-470d-8dd8-ce9365ce4ac2): Axum(Error { inner: Protocol(ResetWithoutClosingHandshake) })
2023-10-21T09:49:28.452512Z  INFO Removing peer: PeerId(30b14bd8-3a85-470d-8dd8-ce9365ce4ac2)
2023-10-21T09:49:28.452760Z  INFO Sent peer remove to: PeerId(fae1d11e-307c-4cfa-92b3-8054264e724c)
2023-10-21T09:49:30.293243Z  WARN Unrecoverable error with PeerId(fae1d11e-307c-4cfa-92b3-8054264e724c): Axum(Error { inner: Protocol(ResetWithoutClosingHandshake) })
2023-10-21T09:49:30.293344Z  INFO Removing peer: PeerId(fae1d11e-307c-4cfa-92b3-8054264e724c)

Local Peer Log

Both of the peer logs are practically identical (from what I can tell).

2023-10-21T09:47:18.858288Z  INFO ace_of_the_heavens::network::session: connection to matchbox server: ws://56.135.57.45:2124/
2023-10-21T09:47:18.899220Z  INFO webrtc::peer_connection: signaling state changed to have-remote-offer    
2023-10-21T09:47:18.899285Z  INFO webrtc_ice::mdns: mDNS is using 224.0.0.251:5353 as dest_addr    
2023-10-21T09:47:18.899540Z  INFO webrtc_mdns::conn: Looping and listening Ok(224.0.0.251:5353)    
2023-10-21T09:47:18.899827Z  INFO webrtc_ice::agent::agent_internal: [controlled]: Setting new connection state: Checking    
2023-10-21T09:47:18.899865Z  INFO webrtc::peer_connection: ICE connection state changed: checking    
2023-10-21T09:47:18.899887Z  WARN webrtc_ice::agent::agent_internal: [controlled]: pingAllCandidates called with no candidate pairs. Connection is not possible yet.    
2023-10-21T09:47:18.899915Z  INFO webrtc::peer_connection: signaling state changed to stable    
2023-10-21T09:47:18.900103Z  WARN webrtc_ice::agent::agent_internal: [controlled]: pingAllCandidates called with no candidate pairs. Connection is not possible yet.    
2023-10-21T09:47:18.900486Z  WARN webrtc_ice::agent::agent_gather: [controlled]: could not listen udp fd::: io error: Cannot assign requested address (os error 99)    
2023-10-21T09:47:18.900522Z  WARN webrtc_ice::agent::agent_gather: [controlled]: could not listen udp fe:80::: io error: Cannot assign requested address (os error 99)    
2023-10-21T09:47:18.900615Z  WARN webrtc_ice::agent::agent_internal: [controlled]: pingAllCandidates called with no candidate pairs. Connection is not possible yet.    
2023-10-21T09:47:18.900739Z  WARN webrtc_ice::agent::agent_internal: [controlled]: pingAllCandidates called with no candidate pairs. Connection is not possible yet.    
2023-10-21T09:47:18.912168Z  WARN webrtc_ice::agent::agent_gather: [controlled]: could not get server reflexive address udp6 stun:stun.l.google.com:19302: io error: Network is unreachable (os error 101)    
2023-10-21T09:47:18.913876Z  WARN webrtc_ice::agent::agent_gather: [controlled]: could not get server reflexive address udp6 stun:stun1.l.google.com:19302: io error: Network is unreachable (os error 101)    
2023-10-21T09:47:48.989217Z  WARN webrtc_ice::agent::agent_internal: [controlled]: Failed to close candidate udp4 host 192.168.43.26:57187: the agent is closed    
2023-10-21T09:47:48.989243Z  WARN webrtc_ice::agent::agent_internal: [controlled]: Failed to close candidate udp4 host 192.168.43.27:34060: the agent is closed    
2023-10-21T09:47:48.989251Z  WARN webrtc_ice::agent::agent_internal: [controlled]: Failed to close candidate udp4 srflx 46.114.176.114:27431 related 0.0.0.0:54440: the agent is closed    
2023-10-21T09:47:48.989262Z  WARN webrtc_ice::agent::agent_internal: [controlled]: Failed to close candidate udp4 srflx 46.114.176.114:28872 related 0.0.0.0:40121: the agent is closed    
2023-10-21T09:47:48.989271Z  WARN webrtc_ice::agent::agent_internal: [controlled]: Failed to close candidate udp6 host :::46363: the agent is closed    
2023-10-21T09:47:48.989280Z  INFO webrtc_ice::agent::agent_internal: [controlled]: Setting new connection state: Failed    
2023-10-21T09:47:48.989321Z  INFO webrtc::peer_connection: ICE connection state changed: failed    
2023-10-21T09:47:48.989331Z  INFO webrtc::peer_connection: peer connection state changed: failed

Note that after this point nothing happens anymore. I let it run for some minutes but nothing happend on either client or server side.

Remote Peer Log

2023-10-21T09:47:17.398532Z  INFO ace_of_the_heavens::network::session: connection to matchbox server: ws://56.135.57.45:2124/
2023-10-21T09:47:18.872610Z  INFO webrtc_ice::mdns: mDNS is using 224.0.0.251:5353 as dest_addr    
2023-10-21T09:47:18.872823Z  INFO webrtc_mdns::conn: Looping and listening Ok(224.0.0.251:5353)    
2023-10-21T09:47:18.873005Z  INFO webrtc::peer_connection: signaling state changed to have-local-offer    
2023-10-21T09:47:18.873150Z  WARN webrtc_ice::agent::agent_gather: [controlled]: could not listen udp fe:80::: io error: Cannot assign requested address (os error 99)    
2023-10-21T09:47:18.903003Z  WARN webrtc_ice::agent::agent_gather: [controlled]: could not get server reflexive address udp6 stun:stun.l.google.com:19302: io error: Network is unreachable (os error 101)    
2023-10-21T09:47:18.904533Z  WARN webrtc_ice::agent::agent_gather: [controlled]: could not get server reflexive address udp6 stun:stun1.l.google.com:19302: io error: Network is unreachable (os error 101)    
2023-10-21T09:47:18.910266Z  INFO webrtc::peer_connection: signaling state changed to stable    
2023-10-21T09:47:18.910381Z  INFO webrtc_ice::agent::agent_internal: [controlling]: Setting new connection state: Checking    
2023-10-21T09:47:18.910454Z  INFO webrtc::peer_connection: ICE connection state changed: checking    
2023-10-21T09:47:18.910478Z  WARN webrtc_ice::agent::agent_internal: [controlling]: pingAllCandidates called with no candidate pairs. Connection is not possible yet.    
2023-10-21T09:47:48.955580Z  WARN webrtc_ice::agent::agent_internal: [controlling]: Failed to close candidate udp4 host 172.17.0.1:33784: the agent is closed    
2023-10-21T09:47:48.955600Z  WARN webrtc_ice::agent::agent_internal: [controlling]: Failed to close candidate udp4 host 192.168.178.30:58126: the agent is closed    
2023-10-21T09:47:48.955605Z  WARN webrtc_ice::agent::agent_internal: [controlling]: Failed to close candidate udp4 host 192.168.122.1:40218: the agent is closed    
2023-10-21T09:47:48.955609Z  WARN webrtc_ice::agent::agent_internal: [controlling]: Failed to close candidate udp4 srflx 56.135.57.45:36014 related 0.0.0.0:36014: the agent is closed    
2023-10-21T09:47:48.955612Z  WARN webrtc_ice::agent::agent_internal: [controlling]: Failed to close candidate udp4 srflx 56.135.57.45:37775 related 0.0.0.0:37775: the agent is closed    
2023-10-21T09:47:48.955616Z  WARN webrtc_ice::agent::agent_internal: [controlling]: Failed to close candidate udp6 host :::40618: the agent is closed    
2023-10-21T09:47:48.955619Z  INFO webrtc_ice::agent::agent_internal: [controlling]: Setting new connection state: Failed    
2023-10-21T09:47:48.955650Z  INFO webrtc::peer_connection: ICE connection state changed: failed    
2023-10-21T09:47:48.955656Z  INFO webrtc::peer_connection: peer connection state changed: failed

Debuggin with Curl

I also tried to curl the server with both local and remote. In both cases I got the same (positive) result. Note that I am using a different port here, but both have been forwarded in the same manner. I also tried the 3536 port in the game and it got me the same results.

Local Curl Log

local-peer$ curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: 21" "http://56.135.57.45:3536"
HTTP/1.1 101 Switching Protocols
connection: upgrade
upgrade: websocket
sec-websocket-accept: I/NEPdKfVBOgESWYT1P625pwr6s=
date: Sat, 21 Oct 2023 10:03:18 GMT

�5{"IdAssigned":"303d3925-8aaa-44a5-9b27-df04300322c7"}

Remote Curl Log

remote-peer$ curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: 21" "http://56.135.57.45:3536"
HTTP/1.1 101 Switching Protocols
connection: upgrade
upgrade: websocket
sec-websocket-accept: I/NEPdKfVBOgESWYT1P625pwr6s=
date: Sat, 21 Oct 2023 10:06:49 GMT

5{"IdAssigned":"704906c1-40ac-41e1-9e68-ca396e46ce24"}

I have also a dummy HTML script on the server and have forwarded port 80 and I can successfully curl the html.index file. That's why I highly doubt that port forwarding is the issue.

PraxTube commented 1 year ago

I tested two more things:

Peers on the same Remote Network

This was successful! I had two peers on the same remote network connecting to the server. This worked without any issues.

Peers on different Remote Networks

This gave me the same errors as the attempt above.

johanhelsing commented 1 year ago

I think that sometimes, it's just not possible to pierce through specific routers, setups, and you have to resort to a relay server (TURN).

You could try a couple of webrtc test sites and see if they successfully get a direct connection, between the peers you have trouble with, if they do, then it's hopefully something we can fix on our end.

PraxTube commented 1 year ago

Ah okay I see, that is actually kinda good news to me. I was a little but under the suspicion that you need more then the matchbox server to facilitate proper online signaling, but the server as it is right now is sufficient to connect peers across different networks? Running the matchbox server on a server with properly forwarded ports should be all you need to do in order for peers to connect to it, right?

I will definitely try out the webrtc tests, might me something with my router, though I also tested the connection of two peers on two different mobile data (using a hotspot) and that also didn't work. I also tested some other matchbox servers besides mine and got the same results, so who knows, will run those tests now. Thanks for your help, very much appreciated :+1:

johanhelsing commented 1 year ago

Running the matchbox server on a server with properly forwarded ports should be all you need to do in order for peers to connect to it, right?

No, there are cases where webrtc on stun alone can't get a direct connection. You'll have to configure it with a TURN server. Those usually require some kind of authentication, otherwise people could exploit it as "free" service.

This is the API you need to use: https://docs.rs/matchbox_socket/latest/matchbox_socket/struct.RtcIceServerConfig.html

You could use xirsys.com free tier for testing development with a turn server, or run an instance of coturn if you have a place to easily/cheaply run docker containers.

EDIT: And sorry this isn't more clearly explained in the docs... Ideally, we'd have an example that uses TURN with a captcha or something.

PraxTube commented 1 year ago

Ahh okay I see, that makes a lot of sense. I will have a look at that.

EDIT: And sorry this isn't more clearly explained in the docs... Ideally, we'd have an example that uses TURN with a captcha or something.

No worries, glad you took your time to help me out :)

PraxTube commented 1 year ago

It works now 🎉 I had to setup a TURN server which I struggled a little bit with. In future versions this will be easier, but there are some pitfalls right now. More info in https://github.com/PraxTube/ace-of-the-heavens/issues/32.

simbleau commented 1 year ago

@PraxTube Congrats. If any ways we can improve matchbox, please submit issues. Big picture items for TURN servers.

PraxTube commented 1 year ago

Thanks <3. It's mostly that there are some bugs that are already fixed in main but aren't in the latest version yet. So you either have to use main or create a fork with the necessary PR merged into it.

If I find any ways to improve this project, I will of course report them :+1: