Implement ICE / TURN / STUN for forwarding packets through NAT

[fire]

Describe the project you are working on:

Unreleased project.

Describe the problem or limitation you are having in your project:

Allow using a ICE server to traverse NAT after UPNP fails.

I'm not able to work on this, but feel free to take on the challenge.

Describe how this feature / enhancement will help you overcome this problem or limitation:

If UPNP fails, one still wants to still be able to connect to the network.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:

https://en.wikipedia.org/wiki/Interactive_Connectivity_Establishment

Describe implementation detail for your proposal (in code), if possible:

• Extract https://github.com/creytiv/re/tree/master/src/ice
• Make it a Godot Engine core module.
• Openssl needs to be stripped and if HTTPS is required, mbedtls will be need to used.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

Networking is not exposed in script.

Is there a reason why this should be core and not an add-on in the asset library?:

This is a core feature.

Other

@Faless is able to help. This may be a good Google Summer of Code proposal.

[Faless]

I won't be able to work on this in the near future, and I don't think I'll have time to mentor this work for GSoC (unless the student is already familiar with Godot and libre), but if specific questions arise I'll gladly answer them.

On Mon, Feb 3, 2020, 16:59 K. S. Ernest (iFire) Lee < notifications@github.com> wrote:

Describe the project you are working on:

Unreleased project.

Describe the problem or limitation you are having in your project:

Allow using a ICE server to traverse NAT after UPNP fails.

I'm not able to work on this, but feel free to taken on the challenge.

Describe how this feature / enhancement will help you overcome this problem or limitation:

If UPNP fails, one still wants to still be able to connect to the network.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:

https://en.wikipedia.org/wiki/Interactive_Connectivity_Establishment

Describe implementation detail for your proposal (in code), if possible:

• Extract https://github.com/creytiv/re/tree/master/src/ice
• Make it a Godot Engine core module.
• Openssl needs to be stripped and if HTTPS is required, mbedtls will be need to used.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

Networking is not exposed in script.

Is there a reason why this should be core and not an add-on in the asset library?:

This is a core feature.

Other

@Faless https://github.com/Faless is able to help. This may be a good Google Summer of Code proposal.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/godotengine/godot-proposals/issues/434?email_source=notifications&email_token=AAM4C3RJCBAKICDHIPAIXUTRBA5PXA5CNFSM4KPHUQ2KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IKUK3KA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM4C3XLR7XRSWX2VCBVIMLRBA5PXANCNFSM4KPHUQ2A .

[henriquelalves]

How forwarding packets using NAT can be solved in-engine? I thought most STUN solutions were mostly server-side, and this can be implemented using a NodeJS server or any other simple server architecture. Am I missing a pre-requisite necessary on Godot to make it work?

Some years ago I tried a NAT hole-punching solution to play Godot Bomberman-Demo over the internet here: https://github.com/henriquelalves/Godot-Bomber-matchmaking-server. I don't quite remember where I left the demo off, so I'm unsure it's still working on 3.2, but I could take a look on it.

[Faless]

I thought most STUN solutions were mostly server-side

But you also need client side support

Am I missing a pre-requisite necessary on Godot to make it work?

We'd like to have STUN/TURN client support in godot to use those protocols. A STUN server is usually very very cheap to rent (in some cases free) so it could be a good option for those that are okay with p2p and don't want to bother maintaining a server infrastructure.

[udit]

Will this be added to 4.0?

[fire]

Not currently planned to be added to Godot Engine 4.0. Feel free to submit a pull request. ICE/TURN/STUN is still wanted.

[henriquelalves]

Not sure if this can be helpful, but since Godot has WebRTC support, I implemented a P2P game in the browser; I described its architecture here: https://write.as/perons/creating-a-peer-to-peer-snake-game-with-godot-webrtc.

So technically the engine does already have ICE/TURN/STUN support as long as you are using WebRTC; I'm not sure whether Godot's WebRTC works on native devices, or if it only works on Web Browsers.

[dsnopek]

I'm not sure whether Godot's WebRTC works on native devices, or if it only works on Web Browsers.

There's an official GDNative plugin that allows WebRTC to work on desktop builds for Windows, Linux and MacOS, as well as a PR that some have reported allowing it work on Android:

https://github.com/godotengine/webrtc-native

I've used WebRTC with STUN in desktop games and works great :-)

[dsnopek]

I started working on a pure GDScript implementation of STUN (it's currently Godot 3.x):

https://gitlab.com/snopek-games/ice-godot/-/blob/main/StunClient.gd

I'd like to eventually expand it to do TURN and ICE, and port it to work with Godot 4.x.

Given how simple STUN, TURN and ICE are, I don't know that they really need to be done in C++ - doing this in GDScript might be just fine. But we may need some minor changes in ENet to take advantage of TURN.

I haven't actually tried implementing TURN yet, but based on reading RFC8656, I think we'd need the ability to prepend some data to every message that it sends. There's two message formats for TURN: (1) that needs a proper STUN header, and (2) the "ChannelData message" format where we just need to prepend 4 extra bytes to each message. The latter seems like it'd be best for our use-case. If ENetConnection could have some API that let us set a PackedByteArray that would be written before every message, I think that could work? I haven't dug into ENet to see if it'll let us put some data before whatever its message format is...

EDIT: I poked around the ENet code a little bit, and I think I see how we could add the extra 4 bytes to the ENet messages. It looks like godot.cpp in the 'enet' module defines the enet_socket_send() function that ENet actually uses, as well as the ENetUDP socket class. I'm not sure how to wire it all together, but if we could somehow set the data to prepend on the ENetUDP socket object, then it could do the actual prepending. @Faless: Does that seem sane?

[Faless]

@dsnopek first of all, really nice work :medal_sports: !

Given how simple STUN, TURN and ICE are, I don't know that they really need to be done in C++ - doing this in GDScript might be just fine. But we may need some minor changes in ENet to take advantage of TURN.

If they really are simple to maintain, given how widespread they are, I think it would totally justify having them in the engine. We might include them in the upnp module, especially since IIRC ICE does have a section for local port discovery.

I poked around the ENet code a little bit, and I think I see how we could add the extra 4 bytes to the ENet messages. It looks like godot.cpp in the 'enet' module defines the enet_socket_send() function that ENet actually uses, as well as the ENetUDP socket class.

Yes, you are correct, the godot.cpp is our custom "platform socket" implementation for ENet. Adding 4 bytes there shouldn't be too much of a problem for plain UDP, quite trickier for DTLS where we will want to double encrypt, something I fear will require quite some refactoring in terms of the DTLS classes.

[lyuma]

This is absolutely amazing! A single-file pure-GDscript implementation of STUN is a really good start, and should allow public IP discovery. The ICE part would be important for piercing firewalls on the existing Enet implementation so that could be really nice to have, and that could be done in GDScript.

While STUN and ICE are pure GDScript, TURN might make more sense to integrate closely with C++ and ENet, since it's probably closer to the critical path in terms of performance... but a pure GDScript approach would also get the job done.

I think C++ TURN implementation might be more important for game servers since they have to handle far more packets per second and performance is more critical

[dsnopek]

Thanks! :-)

@Faless:

Yes, you are correct, the godot.cpp is our custom "platform socket" implementation for ENet. Adding 4 bytes there shouldn't be too much of a problem for plain UDP

Ok, great, thanks for the sanity check!

@lyuma:

While STUN and ICE are pure GDScript, TURN might make more sense to integrate closely with C++ and ENet, since it's probably closer to the critical path in terms of performance... but a pure GDScript approach would also get the job done.

I think C++ TURN implementation might be more important for game servers since they have to handle far more packets per second and performance is more critical

Based on the design of TURN (at least insofar as I understand it currently - I'm sure I'll understand it way better once I try to implement it :-)) I don't think implementing TURN in C++ would really improve performance of the critical path at all. TURN is just some additional attributes and message types for STUN. The most important message type is ALLOCATE, which asks the TURN server to setup a relay to a particular destination, and it responds with the address and port (and some other info) for this "allocation" back to the client. The client can then start sending messages to that address and port, and it just needs to prepend a 4-byte ID to each message (since the TURN server can use the same address/port for multiple relay destinations).

Anyway, this means the TURN messages themselves aren't on the critical path. I think we could do the ALLOCATE and REFRESH messages (you need to periodically "refresh" the allocation, because otherwise it'll go away after 10 minutes) in GDScript just fine. The only bit on the critical path is prepending those 4 bytes, which for 'enet' would definitely need to happen in C++.

So, as far as performance, I think we could keep TURN in GDScript without problems. But I do see @Faless's point that it could make sense to put all this stuff in the engine just to make it more easily accessible to developers.

[fire]

Ping when you're at a good pause point, and I can port it to C++ from gdscript.

[Calinou]

@dsnopek Is your STUN implementation ready to be ported to C++? I think this would be a good feature to have in Godot 4.0 to complement the new replication.

[dsnopek]

@Calinou Sorry, I haven't had any time to work on this lately, and I'm not sure when I'll get a chance to go back to it. It does implement STUN, but I never got around to adding TURN (which is primarily just some additional message and attribute types for STUN). But what's there could be used as a basis for a C++ STUN implementation.

Regarding converting to C++ in general: like I mentioned earlier, I understand that it'd be nice to have this in engine to make it more easily accessible to developers, but from a technical perspective, I do think this is a pretty good fit for a library in GDScript:

1. It's not performance critical, all critical network traffic would go over other channels (after STUN/TURN is used to find the right way to get connected)
2. Not everyone will need it, so it's nice to be optional
3. There's a lot of extensions to STUN and TURN (both standardized and not), which are based around adding new message and attribute types. With everything in GDScript, it's relatively easy to implement those extensions if needed. Of course, it'd be possible to architect the C++ version to be extensible too, but it'd also be easy to miss a needed extension point.

Anyway, feel free to move forward in whatever way you feel is best! I wish I had the time to focus on this, but I've managed to get myself pretty over-committed on too many projects :-)

[lyuma]

@hani09876 please don't leave comments on github asking if something is done yet without contributing new information. You've been doing this a lot.

Proposals, such as this one, are open because they have not been implemented yet. If a feature is to be added, it will first go into master, and later it may be backported into 3.x, but this is no guarantee.

Since you're asking about 3.5, know that using ICE/TURN/STUN connections is already possible in this version by using Godot's official WebRTC GDNative plugin. You can download the WebRTC GDNative plugin here: https://github.com/godotengine/webrtc-native and access it in code by using WebRTCNetworking. There are tutorials explaining how to use webrtc for godot networking.

[mikemherron]

Is anyone activley working on this proposal? Is it still a desired engine feature?

I have some free time, 1-1.5 days a week, for the next month or two and would be interested in helping out on this. I'm not that familiar with STUN/TURN, but do have experience implementing UDP relay servers and clients. If no-one is looking at the issue and it's still something that we want to include in the engine, I can spend time over the next few days going through the linked references and @dsnopek 's GDScript implementation and see if it's something I could contribute to.

[dominiks]

I think it would be a great addition for the included high level multiplayer. If you want to get around port forwarding (manual or UPNP) there's only 3rd party solutions like Steam multiplayer or something selfhosted like Nakama. You would still have to host or provide a way for the initial communication but there's a chance not all the traffic will then go over the relay.

[Calinou]

This is something I'd personally like to see, but note that you will still need to host at least a TURN server yourself if you want to support all use cases (STUN doesn't work with more restrictive network topologies). Also, while there are freely usable public STUN servers out there, you may not want to rely on one in the long term. These servers are provided as-is, and may shut down while your game is still alive and running.

Therefore, there's still some kind of self-hosting involved if you want to deliver a complete, polished experience. You will probably need to create and host a master server to give players a dynamic in-game server list anyway :slightly_smiling_face:

[dominiks]

I fully agree. In retrospect my text seems very much lacking that aspect and reads like you only need a signaling server. The large advantage of ICE/TURN/STUN I see is that it is possible that not all traffic of all multiplayer matches has to go through the relay - minimizing the load on your online services.

[mikemherron]

After reading through this thread and the linked resources I have some questions around the context of this propsoal (if this is not the best place to ask, please let me know):

As mentioned, TURN/STUN/ICE are already available if using WebRTC, either provided by the browser when publishing to web or using the WebRTC native plugin on other platforms. Is there a strong reason why, if a user wants robust P2P over NAT, they would not simply use the WebRTC native plugin? Is the motivation to have something native in the engine that can also be used with the ENet peer?

Related to above, I assume that bringing in the same dependencies (libdatachannel, or libjuice directly) in to the core engine to provide TURN/STUN/ICE support is not a desired approach?

It's not totally clear to me how the ICE process would work with the ENet peer and multiple clients. Previous systems I've worked with tend to have every client connect to a relay right off the bat, rather than attempting to work without one. Would the host pessimisticly use a TURN connection as it's only candidate, and then go through the ICE process with each client? Or is this all TBD as part of this proposal?

[Calinou]

Is there a strong reason why, if a user wants robust P2P over NAT, they would not simply use the WebRTC native plugin? Is the motivation to have something native in the engine that can also be used with the ENet peer?

WebRTC has additional complexity and performance overhead compared to ENet. ENet also doesn't require bothering with GDExtension setup and potential issues, since it's a core engine feature.

I've seen reports of latency being higher with WebRTC compared to ENet, though we need a test project to be able to independently reproduce this.

[mikemherron]

I started working on a pure GDScript implementation of STUN (it's currently Godot 3.x):

https://gitlab.com/snopek-games/ice-godot/-/blob/main/StunClient.gd

@dsnopek as a starting point for this, I was going to fork your StunClient, port it to Godot 4 and then look at adding TURN support, but there is no license on your repo. Would you be up for adding a license so I can fork and modify?

[dsnopek]

@mikemherron Sure, I added a LICENSE.txt (MIT license). However, I did do an initial Godot 4 port about a year ago:

https://gitlab.com/snopek-games/ice-godot/-/blob/godot-4/StunClient.gd

It probably needs some updates for the current state of Godot 4.0, though :-)

[mikemherron]

Quick update on this: I forked @dsnopek 's fantastic work on supporting STUN in GDScript and have the beginnings of a TURN implementation: https://github.com/mikemherron/ice-godot

All it does for now is make a succesful ALLOCATE call (I'm testing locally using PION TURN server), but I have a lot more time over the next 2 weeks to build it out more fully. Once most of TURN is implemented, my plan is to "hack" about with a local fork of the engine and try to get ENet Multiplayer relaying packets through the TURN server. From that point, there will hopefully be enough information to start forming a more concrete proposal for how this could be supported in the engine.

[dsnopek]

@mikemherron Awesome work so far!

For when you get to hacking the engine: I think if we added a the ability to ENet to let us prepend 4 bytes to every message, we could use the "ChannelData message" format to route the packets to the right place.

[mikemherron]

Another quick update on this. The WIP TURN client is now able to create channels and send and recieve data using the ChannelData mechanism. The code is super rough right now and needs a lot of tidying up. It is also nowhere near supporting the full spec, but does enough as a workable POC.

I've not looked at ICE at all, not considered DTLS or a mix of STUN/TURN peers, just trying to get a basic working example with all-TURN plain UDP as a first step.

I spent some time looking through the multiplayer code and tried to plan out what would need changed for ENet to route messages through TURN. This is what I've come up with, but would appreciate any feedback or guidance. The code is all new to me so there's a pretty good chance I've misunderstood some things 😅

• When creating the client and server sockets using ENetMultiplayerPeer.create_server/create_client, the underlying UDP sockets need to listen on the same local port that was used to establish the TURN connection. This should be simple as both methods alread support specifying the local port.
• The code that manages the TURN connection will probably need to first disconnect to free the port, but then send future "management" messages (mostly making REFRESH calls to keep the allocation active) to the TURN server through the socket that is created by ENet. It's not clear how best to do this - I'm wondering if connect_shared_socket on PacketPeerUDP could potentially allow ENetMultiplayerPeer to return a PacketPeerUDP to GDScript that acesses the same underlying NetSocket? I guess this would have to be used with care as reading messages could prevent them from being handled by Enet, but might be a good solution for sending messages.
• As @dsnopek said, enet_socket_send in thirdparty/enet/godot.cpp will need to prefix messages with the ChannelData header. It will also need to look up the correct channel number (that's the TURN channel, not the ENet channel) to use for the header. If a callback function could be injected here that would be straightforward to do, but (without much knowledge of Godot internals) it seems like that is fairly down in the deep dark depths, and not much of the surrounding classes are exposed to GDScript. I'm not sure how feasible this is but will look in to it more.
• The corresponding enet_socket_receive function will need to do the opposite and strip the ChannelData header before the usual processing takes place. Again, if a callback function could be injected that would be straightforward.
• The enet_socket_receive function will also need to be able to hand-off processing of other TURN messages (like REFRESH responses) to the TURN management code and prevent them from being processed by ENet. I think this could be done by having the call back return 0/false/null to indicate the message should not processed.

As an alternative to all this, I did wonder if a simpler integration could be to have a setup that acted like a sort of "local proxy". Once the TURN allocation has been established, the turn code would open a new unconnected local "proxy" socket. ENET Servers and Clients would send messages to that local socket. The turn code would then read from that socket and add the required TURN formatting before sending on to the relay. Incoming relayed packets would have the TURN formatting removed and then sent from the "proxy" socket to the client/server socket where ENet would read from.

It needs some more thought (and would introduce an extra local "hop" so might not be viable from a latency POV) but if it worked would mean TURN could be used without major changes to the existing code.

I'll carry on my investigation and see how far I can get...

[mikemherron]

I managed to make some progress with the proxy concept and have a working prototype sending ENet traffic across a public TURN relay:

https://user-images.githubusercontent.com/15673068/230481898-3cc675f4-c03c-4c84-a043-bf81e4d2c200.mp4

The diagram below shows what's happening behind the scenes:

The proxy code itself is fairly straightforward as is the configuration once the TURN allocation and channels have been set up. I'm going to try and apply to this one of the multiplayer examples and see how it performs once I've tidied up the code.

It's all still GDScript right now, but I think this approach could also be implemented in C++? It is appealing as it avoids the need to "pass around" the underlying socket to continue the turn allocation management, it also feels nicely decoupled and avoids needing to introduce structural changes to ENet. For users that need DTLS it could be implemented at the TURN level (ENet would stay plain UDP). Downsides could be performance, and spawning a load of sockets that aren't strictly needed might increase security risk?

[fire]

I'm confused if I use enet in dtls mode, godot engine's enet dtls passes through with no special effort?

[mikemherron]

I'm confused if I use enet in dtls mode, godot engine's enet dtls passes through with no special effort?

Yes it does, but if you needed DTLS at the TURN layer too (I believe some console platforms require total encryption), the TURN messages would also need DTLS applied so there is some redundancy there. It may be simpler (but not required) to just apply DTLS once at the TURN layer in that case. There was also a comment earlier in this issue from @Faless that suggested doing the double encrypt might require some refactoring - but that was from Dec 2021 so may no longer be the case.

[fire]

I prefer that game client DTLS is encrypted to the DTLS game server. I don't mind that it may be double encrypted at the turn level.

I'm hyped!

[fire]

@mikemherron I had some trouble testing this with https://www.metered.ca/tools/openrelay/. Do you know what the realm is?

Erroring with 401 code.

[mikemherron]

@mikemherron I had some trouble testing this with https://www.metered.ca/tools/openrelay/. Do you know what the realm is?

Erroring with 401 code.

The realm is metered.ca for that service - however there was a bug in the code for parsing the error reason that was preventing other attributes from being parsed. I've pushed a fix for that - I can now see the client sends the authenticated allocation request, but the servers are not responding after that. I will look in to it over the next few days and try figure out why - there are 2 different credential mechanisms with TURN (short term and long term) - the client just now only implemenents long term, it could be the server expects short tern. It could also be the packet is just getting lost as the client is not doing any retries like it should (it's missing a lot of stuff right now)

(EDIT: for my own testing I've been deploying PION server to a GCP VM, I've not tested with other implementations at all. Given how little of spec is implemented there will likely be issues at this stage with other services).

If the socket proxy approach seems viable I think it could still be worth finding an existing, minimal, TURN/ICE implementation in C++ and integrating that, then layering the proxy code on top. In any case, I'll continue to expand the existing GDScript implementation for now.

[Faless]

@mikemherron that is really awesome :fireworks: . I wonder if it still makes sense to write a module in C++ and then (optionally) use it in thirdparty/enet/godot.cpp to avoid a bunch of packet re-allocations. It also seem pretty flexible so it could be used by other network peer, and it doesn't seem to be a lot of code.

[fire]

@mikemherron I think https://github.com/libnice/libnice/blob/master/COPYING may work since we can choose the MPL license for ice and stun support. However a second look with it, has a scary amount of dependencies so it may still be better to do our own code.

[mikemherron]

@mikemherron that is really awesome 🎆 . I wonder if it still makes sense to write a module in C++ and then (optionally) use it in thirdparty/enet/godot.cpp to avoid a bunch of packet re-allocations. It also seem pretty flexible so it could be used by other network peer, and it doesn't seem to be a lot of code.

Yes definitely, I think it would be far more suited to C++. The current code is totally unoptimized. There could possibly be some sort of PacketRelay abstraction, and then a TurnPacketRelay implementation perhaps. It would be nice if then users could configure enet (and possibly other peers) with a specific Relay. I'd still quite like to see the GDScript POC working with a more realistic example; once I can prove that I'll sketch out how it could look in C++ 👍

[mikemherron]

@mikemherron I think https://github.com/libnice/libnice/blob/master/COPYING may work since we can choose the MPL license for ice and stun support. However a second look with it, has a scary amount of dependencies so it may still be better to do our own code.

Yes I think you may be right. It also contains it own socket abstractions etc that Godot already has, it might be a lot of bloat. I have at least a few more weeks I can spend on this so will continue to build out the GDScript version in the meantime - that will give a good indicator of much work a C++ implementation would be.

[fire]

Actively supporting the pion server and coturn (metered.ca) is a good step. Once that is done, let me know when you're ok with me or others porting the code to c++.

Typically not that hard to mechanically port gdscript to c++, but changing the software design is more difficult so I want to keep it in gdscript until the design flaws are fixed. We can push for optimization in the c++ port.

I am interested in testing a server based on libnice but I don't have a server implementation yet. https://github.com/membraneframework/ex_libnice

Edited: Can you summarize in a todo list of what steps are pending for https://github.com/mikemherron/ice-godot

[mikemherron]

Actively supporting the pion server and coturn (metered.ca) is a good step. Once that is done, let me know when you're ok with me or others porting the code to c++.

Typically not that hard to mechanically port gdscript to c++, but changing the software design is more difficult so I want to keep it in gdscript until the design flaws are fixed. We can push for optimization in the c++ port.

I am interested in testing a server based on libnice but I don't have a server implementation yet. https://github.com/membraneframework/ex_libnice

Edited: Can you summarize in a todo list of what steps are pending for https://github.com/mikemherron/ice-godot

My goal for now is just to implement the absolute minimum set of TURN features required to prove the socket proxy approach is viable. This week I'll aim to take one of the multiplayer samples and integrate with the POC code to do that. I'm not sure what the todo list is, it's more a case of trying stuff and fixing it up if it doesn't work. I wasn't planning to write the full TURN/ICE implementation in GDScript - there are so many existing implementations out there it feels like implementing the various specs is low-risk, and there wouldn't be much gained writing it all in GDScript only to port to C++, provided the structure of the C++ implementation is well understood before starting.

Once I've got the POC working with a more realistic example I was going to put together an outline proposal of how a C++ implementation could look in terms of classes, integration with existing code and the interface exposed to users (I'll need to understand better how ICE fits in to the picture at this point, and also how it will work with the mesh configuration and not just server-client). Once that had some sort of maintainer buy-in, I was planning to start work on the C++ implementation - help on that would be fantastic (also more than happy to accept PRs on the POC code)

[mikemherron]

I spent some time today quickly adapting the Godot 4.0 Bomberman example to use POC TURN client and proxy socket approach - it is functional, but gets super laggy the longer the game goes on:

https://user-images.githubusercontent.com/15673068/231242668-02c6fb59-56ca-44f4-aa4d-f3df076c2d85.mp4

The round trip time starts off OK-ish but then just goes up and up. Another 30s after I stopped recording it was almost 1s. Ping time to the TURN server is 30ms from my machine. Packet loss seems high, but I'm not totally sure how to read this w.r.t. the packet loss scale constant in ENet?

I'd be surprised if PION server was causing such issues, so my guess is the problems are with the existing TURN client or Proxy code. I made a quick change to process all packets each frame (rather than just 1) which seemed to help a bit but still gets worse the longer the game goes on. I'm not super familiar with ENet so not sure if anyone has any suggestions or ideas where this behaviour comes from or what this is typically a symptom of?

I will look in to it (I'm sure there is massive scope for improvement given the early stage of the code) but any pointers appreciated...

[fire]

Hypothetical question. Do you think you can make it fail twice as quickly? Do you think it's related to framerate? Trying to binary search the code bases of pion and godot engine.

Does turn have a sequence number or reliable udp packets? Like enet channels

[Agredo]

I spent some time today quickly adapting the Godot 4.0 Bomberman example to use POC TURN client and proxy socket approach - it is functional, but gets super laggy the longer the game goes on:

turn-bomberman.mp4 The round trip time starts off OK-ish but then just goes up and up. Another 30s after I stopped recording it was almost 1s. Ping time to the TURN server is 30ms from my machine. Packet loss seems high, but I'm not totally sure how to read this w.r.t. the packet loss scale constant in ENet?

I'd be surprised if PION server was causing such issues, so my guess is the problems are with the existing TURN client or Proxy code. I made a quick change to process all packets each frame (rather than just 1) which seemed to help a bit but still gets worse the longer the game goes on. I'm not super familiar with ENet so not sure if anyone has any suggestions or ideas where this behaviour comes from or what this is typically a symptom of?

I will look in to it (I'm sure there is massive scope for improvement given the early stage of the code) but any pointers appreciated...

Thank you for your example. I am trying to digging into it but I did not find any turn server thats works with your code. Do you have any tips? I tried turn:openrelay.metered.ca:80 with openrelayproject and metered.ca as realm.

Is there any thats works to test it?

Thank you :)

[mikemherron]

@Agredo I was using PION turn server locally and on a GCP VM to test. Unfortunatley I never tested with another TURN server, I was kind of racing to get something working end to end to prove the concept before going back and "doing it properly".

If you wanted, you could also try run the debug panel from the original repo used to get TURN working: mikemherron/ice-godot the TURN/ICE code should the same as the bomberman demo. The debug panel let's you connect to a TURN server and see requests/responses that might help pinpoint what's going on.

I will set aside some time this evening and try with openrelay - it should be very possible, PION is fairly standard, no doubt something I've missed from the spec. I have a new full-time role so sadly can't dedicate much time here but great to see someone else taking a look!

[mikemherron]

I manged to get test application from the original repo working with metered.ca using the following settings:

Address: 216.39.253.22 Port: 80 Username: username from metered.ca dashboard Password: password from metered.ca dashboard Realm: metered.ca

https://github.com/godotengine/godot-proposals/assets/15673068/a58f2052-75d9-472b-bebb-12c399eb3edb

(Sorry the video is really choppy, not sure what's going on...btw I also removed my metered credentials after doing the video 😄 )

A cocuple of things to note...

• To get the actual TURN server address metered want you to use their API - I just called it once from the JS conole and grabbed the first relay server
• The actual server was something like 'a.relay.metered.ca' but for some reason Godot couldn't resolve that...I had to manually pass the IP 🤔 Not sure why...
• The debug panel shows an error about round tripping -the code attempts to deserialise then reserialise messages to make sure they are the same. In this case, the allocation response is not the same - this means there might be one or more attributes from the spec not yet implemented in the code
• For some reason (it's a bit hard to see in the video) the channel data doesn't make it through on the first attempt, only once I've tried to send in the other direction - I didn't see this behaviour with PION, not sure if it's something specific to the TURN server metered are using or a bug in the code...
• I couldn't get the bomberman example to work with metered...the client could never reach the server - the channel data messages were never received. My first suspicion is the point above...I'd perhaps try send some dummy channel data message first before initiating the ENet proxy and see if that helped.

As you can see it's still at a super early stage. Apologies I couldn't leave it a little more polished and easier to pick up - but if you do manage to make some progress I'd love to hear more.

[krazyjakee]

@mikemherron Sure, I added a LICENSE.txt (MIT license). However, I did do an initial Godot 4 port about a year ago:

https://gitlab.com/snopek-games/ice-godot/-/blob/godot-4/StunClient.gd

It probably needs some updates for the current state of Godot 4.0, though :-)

How do I get from your example to allowing 2 peers to connect and use MultiplayerSpawners and MultiplayerSynchronizers without port forwarding?