NetworkedMultiplayerSteamP2P

[ghost]

I have read this issues GodotSteam/GodotSteam#145 GodotSteam/GodotSteam#124 and saw this repo https://github.com/JDare/GodotSteamHL but I still don't quite understand what exactly is the problem with integrating HLAPI with Steam's networking sockets directly with a dedicated peer on a module-level. Most of what I came across the internet (including mentioned repo) is some basic lobby system setups to pass arbitrary streams of data in-between by using self-made protocols e.t.c. which is not that hard to set up in the first place. But what about the usage of Godot's multiplayer API (slave/puppet/RPC stuff/position sync/attribute sync/etc)? All that essentially needs to be done is the replacement of data sends-receive/ connect functionality from ENet to the SteamAPI, while still having a fully backed by the engine serialization/synchronization functionality provided by high-level API. @Gramps I would really appreciate some feedback on this since all of the classes mentioned in those issues as far as I know are already in place. What would be the most reasonable way to implement something like this on the current version of the module, since most of the required functionality is now accessible to the users of GodotSteam?

[Gramps]

My understanding of networking is pretty basic. And I will also note that the newer Steam networking classes will be replacing the current networking class and P2P stuff. The newer stuff is still in beta, last I checked, and not fully implemented in GodotSteam yet.

That being said, I have still yet to investigate mixing Godot's HL networking with Steam's fully; way too much on my plate still. My off days usually get overwhelmed by one random thing or another. I think that mixing the two might just be overkill though; more code than needed to do the job. To me it makes more sense to use one or the other; and if your game is on Steam might as well take advantage of their networking features instead of coding it all yourself in Godot's.

I think one of the original problems was that Godot's relied on IP address where as Steamworks does not; Steam handles all the address and port stuff if you use their functions correctly. I had an example of passing the IP through Steam's invite system but it was kinda janky. My last Godot 2 game did this and it was messy but functional. Also, I think the newer networking stuff does more to expose the IP and port data though. In my personal projects I have started working more with networking and authentication, but strictly using Steamworks functions and no Godot networking. It seems a little more straight-forward than the Godot HL networking; but I may be biased or somewhat uninformed.

Other than in tutorials, nothing from Godot's networking will be mixed into the module; it should stay as close to Steam's native implementation as possible. If I or someone else can properly mix the two and write up a tutorial, it'll get added. Like I said, I plan to tackle it at some point.

[Gramps]

Nonetheless, I have it on my list to review and it'll happen at some point. Just might be a little while.

[ghost]

Thanks for the response! I really really really need this feature for my game, since using custom protocol is not an option for me (I tried going that way, but the overhead that I had due to the complexity of my game made me write more and more of the networking code then of the game itself), I need to make P2P and HLAPI friends somehow (In this context I mean P2P connections, not P2P as an authoritative model). If it matters, I can finance you for your development time spent on this. (I tried hiring someone at Godot's discord server, but no one has responded so far).

[Gramps]

Is there any reason you need to use the Godot HL networking stuff? Other than perhaps re-writing a bunch of things. If making it strictly P2P with no master/slave, that should be easy to do with Steamworks.

[ghost]

I really don't want to reinvent the wheel with position syncing, scene instantiation sync, the scene tree sync, RPC, object ownership and there are just so many other things that need to be synced if I am going to use custom protocol, wherein HLAPI almost everything that I have mentioned comes pretty much out of the box, and is pretty much an officially supported way of handling networking in Godot.

[Gramps]

Yeah, that would probably be a lot of work and a pain in the ass.

[rcorre]

I ended up implementing this using CustomMultiplayer in Godot 3.5. It seems to be working for me as a drop-in replacement for NetworkedMultiplayerEnet, with the exception that you have to manually call poll (which I believe will no longer be a limitation with MultiplayerAPIExtension in godot 4):

https://gist.github.com/rcorre/776926518a962c3cb193e07de1bf61b3

@Gramps would you be intrested in me contributing this to the docs as a gdscript example, or even to GodotSteam itself as a c++ implementation?

You can use it like this:

func _ready() -> void:
    # Could also let tree automatically poll, but I wanted to control the order of polls
    get_tree().multiplayer_poll = false

func _physics_process(_delta: float) -> void:
    if multiplayer.network_peer is NetworkedMultiplayerSteam:
        (multiplayer.network_peer as NetworkedMultiplayerSteam).poll()
    multiplayer.poll()

func start_game():
    var lobby_id := create_or_find_lobby() # However you want to do this
    var peer := NetworkedMultiplayerSteam.new()
    peer.join_lobby(lobby_id)
    get_tree().multiplayer.set_network_peer(peer)
        # now use Godot's RPC system

[Gramps]

Hey there! That is really neat and, I think, in line with what @greenfox1505 has been up to: https://github.com/greenfox1505/GodotSteam/tree/multiplayer-for-godot4

I think merging Steamworks into Godot's networking stuff is a great idea and would love to integrate it into GodotSteam. I've been super busy but my time is opening back up so I can start looking into this stuff more. Seems you and @greenfox1505 have laid most of the groundwork already too!

I'm not sure how much the two differ but maybe we all can start a discussion about it. What say you @rcorre and @greenfox1505?

[greenfox1505]

Multiplayer peers on both Godot3 and Godot4 both will call poll every frame when a peer moves to a connected state. But GodotSteam needs a call back every frame in order to register that you have moved to a connected state.

This is cool though. I'd like to see your code. At some point my project will get merge in here on GodotSteam and if there are some useful things from your version, I'd like to include them.

Edit: SORRY, I just noticed your Gist. OH COOL! You can make multiplayer peers with just GDScript now? That's awesome. That used to not be possible. Looks like you're doing a lot of what I'm doing.

[rcorre]

Multiplayer peers on both Godot3 and Godot4 both will call poll every frame when a peer moves to a connected state

Unfortunately not if you're using gdscript since poll is not virtual on the gdscript side. That's not a problem in c++ though.

It would be great to work together to get this implemented though.

@Gramps would you want an implementation for Godot3, or just Godot4? I steered away from the latter just because I wasn't sure if the APIs were stable.

[Gramps]

I would say both as 3.x is still heavily used but 4.x eventually will be the main use. I'm not sure how hard it is to port to both of them. Since my current game project is coming to a close soon (though I thought it already would have), I need to get more up to speed on all of this soon.

I think 4.x is finally stable enough. I believe someone was also porting @greenfox1505's stuff into Godot 4x.... right?

[rcorre]

Well, I was going to ask if I should bring in my 3.x implementation, and just like that, 4.0 is officially released :laughing:

[Gramps]

Hmm, that I don't know. @greenfox1505 is spearheading this venture so perhaps he'd know the best course of action. I'm down for whatever!

[greenfox1505]

I'm doing my stuff in godot4.0 now. There is an old version that mostly worked for godot3.x, but everything I'm working on now is godot4.

What I'm working on can totally be backported to godot3.x, if you need it. I can help a bit with that, but it won't be a priority for me.

[Gramps]

Having this in 3.x would be awesome but 4.x is the future. Surely we can do both. :D

[rcorre]

What I'm working on can totally be backported to godot3.x, if you need it. I can help a bit with that, but it won't be a priority for me.

Oh, no worries. I already had a working solution for 3.x, and was asking if I should port it to c++ for GodotSteam. But with 4.0 officially being released, I'm just moving to 4.0.

[Gramps]

Well if everyone is focusing on 4.x then I can handle the 3.x porting.

[rcorre]

I managed to get something working for 4.0, here it is in case it's useful: https://github.com/Gramps/GodotSteam/compare/master...rcorre:peer4.0

[greenfox1505]

I will look at this at the end of the week. I have some day job deadlines before GDC and then after that I'm gearing up for my own trip. So I may not have time to review this until at least after that. https://github.com/greenfox1505/GodotSteam/tree/multiplayer-for-godot4 This is my current project space. If you've got some fixes and behaviors I don't, we can absolutely merge that in.

[rcorre]

Thanks @greenfox1505 , no rush at all! I just wanted to get something to the point where I could continue with my game. It looks like our implementations ended up pretty similar.

• Mine's minimal and doesn't replicate much functionality already in Steam, so I didn't add any signals or functions for e.g. creating lobbies. I figured you'd still use Steam to do that.
• Yours has a builtin ping mechanism mine doesn't -- I figured you'd do that via rpc if you wanted one.
• You use generate_unique_id while I use the bottom 31 bits of the AccountID. I suppose both have the same probability of a collision, though generate_unique_id is probably better, because if two players do have a collision, they'd be able to retry and it probably would work. The docs don't actually say the peer_id has to be a uint31 any more (they did in 3.5), but based on the code that still looks true. I asked here: https://github.com/godotengine/godot/issues/74628#issuecomment-1463717863
• Neither of us implemented unreliable ordered rpc (seems doable, but annoying).

Hope GDC is fun!

[greenfox1505]

I liked generating the id because it's unique to the connection. If you loose connection and have to reconnect, you get a new id. It does have the possibility of collision, but so does enet.

The ping functionality I have is purely for keep-alive. A client doesn't register a peer has joined until it gets one ping from it. Then it periodically pings to make sure the connection is still valid. The next step there would be to reestablish the connection on failure.

A lot of Godot's built in peers don't implement various packet types. For example WebSocket only implement ordered reliable. I figure making ordered unreliable to ordered reliable was valid.

[MichaelMacha]

I was just dropping in to suggest something like this, and so many people have beaten me to it!

My big interest is to use Steam, but not be limited to it—eventually, I might want to drop this on Epic, or another game platform, and I need to maintain that flexibility. I also like the idea of people being able to host their own games locally or without relying on a third party network.

I'm also not stoked about dropping down to GDExtension for the sake of it, because I would need to support multiple platforms (including Mac, which I don't have for testing) and also feel that it should be reasonably standardized... So it's kind of a last-resort thing for me. (I have almost no experience with GDExtension.)

That said, I have a plan here. ENet is how Godot generally does local multiplayer. This relies on ENet being set as the peer, so that MultiplayerSynchronizer and MultiplayerSpawner work out-of-the-box. However, if we set the peer to a dummy (Maybe OfflineMultiplayerPeer), and add a new child node that uses exposed GodotSteam functions to synchronize the chosen properties over Steamworks instead of ENet, we will have a drop-in replacement that depends entirely on configuration.

That's what I'm working on at this point, or at least it's my action item list. We would have MultiplayerSynchronizer\SteamSynchronizer, and MultiplayerSpawner\SteamSpawner; both of which would examine properties of their immediate parent and transmit them over Steamworks, while the parents would remain relatively inert.

This is less flashy than implementing MultiplayerPeerExtension, but it's flexible, insulates me from the hardware level, and could easily be swapped out for, say, some future EpicSynchronzier and EpicSpawner or something like that. It would entirely be a question of the game mode. The other advantage is that we have zero guarantee that network game services are all going to provide comparable interfaces.

Assuming this all holds up, maybe it might solve some outstanding issues for GodotSteam too?

[rcorre]

This relies on ENet being set as the peer, so that MultiplayerSynchronizer and MultiplayerSpawner work out-of-the-box

Will MultiplayerSynchronizer and MultiplayerSpawner not work with an arbitrary peer? I haven't tried them yet, but I figured they'd use the rpc system, so as long as you have a MultiplayerPeer that implements RPC they should "just work".

[greenfox1505]

The MultiplayerSpawner and MultiplayerSyncer already work. SteamMultiplayerPeer is only a transport layer. Under the hood, they use the same transport paths as RPCs.

[MichaelMacha]

@greenfox1505 I'm sorry, I seem to have totally missed that you're extending it to include a Steam peer. It was a lot to read. Thank you!

I'll try implementing it tomorrow. It's been a little while since GDC, have you made much progress? If it's ready to go, I may dive into it just to get a better feel of how peers work under the hood. (Mostly a multimedia guy, my networking knowledge is tangential.)

Does it need any documentation written, or is there anything I could help with coming in late?

[greenfox1505]

https://github.com/greenfox1505/GodotSteam/tree/multiplayer-for-godot4

My repo is here. My plan was to merge this in when I had finished a game jam project using it, proving its soundness. That jam did not go as planned. Now the plan is to create a simple project that will become a proof of concept for most of what I'm trying to accomplish here.

[MichaelMacha]

Sorry your jam didn't work out! That said, I've got an arena shooter going on ENet which I want to publish on Steam, and the ability to simply change the network peer and have it work as usual over Steamworks would be a blast.

I'm going to go ahead and download it, copy my project, and see if I can get this to work with your GodotSteam branch on game ID 480. Is there anything I need to worry about with, say, building for other export templates? I'm running Mint 20.3 Una right now, and I'll also want to be building a Windows executable.

With everything I've done with ENet, I might be able to help with that proof-of-concept project too.

[rcorre]

Is there anything I need to worry about with, say, building for other export templates? I'm running Mint 20.3 Una right now, and I'll also want to be building a Windows executable.

@MichaelMacha I'm also developing on Linux. The normal cross compiling instructions for Godot do not seem to work with GodotSteam. Compilation succeeded, but the resulting binary wouldn't start when friends tried to run it on Windows .

I ended up modifying the Github Actions on my fork to produce builds for me (the GodotSteam actions pull the SDK from a private repo).

I do have a release here I've been using with my peer implementation (I had already started on one before I realized there was work in progress).

[MichaelMacha]

@rcorre Odd, I'll bear that in mind. I had a few issues getting it to compile a moment ago, until I realized that it was also looking for (I think) the tools directory from the Steamworks SDK. When I unpacked everything into the module, it started to work. All the instructions asked for was public and redistributable_bin.

The example application was probably unnecessary, but hey, I get paranoid when I'm arguing with a compiler.

Time for a break, but I'll keep your repo in mind if I hit flak on a build.

[MichaelMacha]

@greenfox1505 Scratch that, I apparently was using the Gramps GodotSteam. On Linux, using your branch, I'm getting a long sequence of incomplete type (generally GetTypeInfo) errors, and ambiguous conversion errors, which prevents the compilation from completing.

It almost sounds like I'm missing a header somewhere, somehow. Most of them are in header files. They all happen when Scons hits steam_multiplayer_peer.cpp. Maybe someone can give me a hint what I'm missing?

I'll drop the compilation report on pastebin for you, if it helps. Everything else has already been built.

Thanks for all your work here.

ADDENDUM: Seems that _bind_methods is having issues (or more likely, I'm having issues understanding it). Tomorrow I'm going to spend a portion of the day learning GDExtension in detail, as this feels like it's a me-problem. I certainly can't think of a more productive time to learn it.

[cridenour]

We're using @greenfox1505's peer (but rebased on a later GodotSteam build) in production with our Early Access game. High level nodes work most of the time - we noticed an issue trying to use a Synchronizer right at the start (we just use RPCs in that case). But that could also be something else going on in our game.

But we've had hundreds of people playing and streaming multiplayer over Steam without issue.

We also offer ENet (just call it Direct IP to players) and Offline - switching between peers is effortless.

This should definitely be the future of GodotSteam's multiplayer approach. Happy to help where I can.

[Gramps]

That's super-cool it is working in a big use-case. What is your game called?

[cridenour]

That's super-cool it is working in a big use-case. What is your game called?

Oh you already know it ha - Drift

[Gramps]

Ah yes! Last I played I was drifting off into the void!

[scriptsengineer]

Although the start of the issue is about P2P, I wanted to update about the state of Multiplayer Peer with GDextension, I'm currently working on branch: https://github.com/expressobits/GodotSteam/tree/gdextension-with-multiplayer in order to peer with godot steam in gdextension.

[Gramps]

This project just keeps getting cooler!

[MichaelMacha]

I'm also working on a GDExtension SteamMultiplayerPeer class, admittedly mostly just to improve my knowledge of GDExtension, peer networking, and brush up on C++. You may end up having to cherry pick from the lot of us, Gramps! It's great to see so much interest.

My goal is to get to the point where arena-style multiplayer is just as easy with GodotSteam as it is with ENet or another default option. Just plug in the peer and let the synchronizer worry about it. I seem to be tripping on a missing pure virtual method, though... could be a bit.

[Gramps]

Yeah, people have been asking about how to do this for quite some time. That's awesome you all are working on it.

[scriptsengineer]

I'm also working on a GDExtension SteamMultiplayerPeer class, admittedly mostly just to improve my knowledge of GDExtension, peer networking, and brush up on C++. You may end up having to cherry pick from the lot of us, Gramps! It's great to see so much interest.

My goal is to get to the point where arena-style multiplayer is just as easy with GodotSteam as it is with ENet or another default option. Just plug in the peer and let the synchronizer worry about it. I seem to be tripping on a missing pure virtual method, though... could be a bit.

Wow I'm glad you're venturing too, currently I'm just making small tweaks, like using enums that already exist on godotsteam.

Although I think the biggest pain is figuring out if there's any way for steamMultiplayerPeer not to duplicate calls or signals, like create_lobby

[MichaelMacha]

@scriptsengineer There's usually an architectural way to work around potential duplicate code. We'll figure something convenient out. I'm thinking maybe something like a SteamMultiplayerPeer constructor that takes the connected lobby ID as a parameter...

The biggest issue I'm looking at is comprehensively understanding GDExtension. I've got a bunch of little things working fine, but extensions of MultiplayerPeer itself insist on being abstract in GDScript. It doesn't look like an abstract class on the C++ end. Am I just overlooking something simple?

[rcorre]

@MichaelMacha I may be misunderstanding your question, but in GDExtension I believe you need to extend MultiplayerPeerExtension instead of MultiplayerPeer.

Check godot-cpp/gen/include/godot_cpp/classes/multiplayer_peer_extension.hpp after building in godot-cpp. You'll see that all the methods are virtual (unlike multiplayer_peer.hpp).

[scriptsengineer]

Wow was answering now that, that's exactly what the @rcorre says,

Another update on the implementation of gdextension is that I intend to redo the work of greenfox using steam sockets and not lobbies, at first lobbies is something of high level that the user must implement something related. See more in discord topic https://discord.com/channels/564589037411893260/1021131693979090994/1115437306086690916

[MichaelMacha]

I believe you need to extend MultiplayerPeerExtension instead of MultiplayerPeer.

Oh Lord... I figured it was something ridiculous. @scriptsengineer Thank you sir; GameDev Stack, Stack Overflow, and the Godot forums had nothing for me on this, and you saw the problem immediately. :-)

On the bright side, chasing this awkward bug has given me a profound understanding of how arbitrary parts of GDExtension work under the hood; so I guess there's that.

My explicit goal is to get the data streams already synchronized over ENet (as an example), without any change of their formatting, to flow over the Steam network instead; I'm trying to finick with the minimum possible amount. My goal is to be able to set up a property to be synchronized, and simply have it pipe through Steam instead of another network. As I understand it, that's most of what Peer orchestrates.

[scriptsengineer]

I believe you need to extend MultiplayerPeerExtension instead of MultiplayerPeer.

Oh Lord... I figured it was something ridiculous. @scriptsengineer Thank you sir; GameDev Stack, Stack Overflow, and the Godot forums had nothing for me on this, and you saw the problem immediately. :-)

On the bright side, chasing this awkward bug has given me a profound understanding of how arbitrary parts of GDExtension work under the hood; so I guess there's that.

My explicit goal is to get the data streams already synchronized over ENet (as an example), without any change of their formatting, to flow over the Steam network instead; I'm trying to finick with the minimum possible amount. My goal is to be able to set up a property to be synchronized, and simply have it pipe through Steam instead of another network. As I understand it, that's most of what Peer orchestrates.

I'd be happy to see this work in progress, my plan is to do just that by converting enet code to steam sockets, just like that.

[MichaelMacha]

@scriptsengineer It's later than heck, but I've successfully gotten an instantiable NullaryMultiplayerPeer going. Thank you. Code is progressing along.

[EvilDraggie]

Y'all are doing great work. I'm currently playing around with @cridenour 's implementation and just now managed to get the first basic things to work, though I seem to have issues with the MultiplayerSpawner not being found? Maybe @cridenour could elaborate a little on how their implementation is used? Might help other peeps too.

I've been working on a co-op space-horror game the past 2 years, using ENET for the multiplayer stuffs which is working very well now. Then I realized that for the Steam version I needed some sort of relaying through Steam to not have to deal with NAT/ports etc so needless to say what you all are doing has been a lifesaver. Now I don't need to rewrite all of my netcode haha. And I can also release the game on other platforms if needed! woah!

Looking forward to an official implementation in GodotSteam in the hopefully near future :)

[cridenour]

though I seem to have issues with the MultiplayerSpawner not being found?

Does lobby/PlayerSpawner already exist when the connection is initiated or are you changing scenes as part of the connection process? I've found it's more reliable have the scene fully loaded before connecting to other players.

[cridenour]

redo the work of greenfox using steam sockets

I had a similar conversation with Faless (who wrote the new Godot nodes) in RocketChat and he agreed that Godot is happier when connections are lower level so it may be a better fit than the Messages API.

[EvilDraggie]

though I seem to have issues with the MultiplayerSpawner not being found?

Does lobby/PlayerSpawner already exist when the connection is initiated or are you changing scenes as part of the connection process? I've found it's more reliable have the scene fully loaded before connecting to other players.

I make the initial connection in the main menu, then load up the lobby. When the lobby is loaded and running I send an RPC to the server that it needs to spawn a new player. I shall try what you suggested :) thanks!

EDIT: it works now! The suggestion worked. I'm a little iffy about connecting after loading, since if the connection somehow fails people have to sit through the loading stuff, and then wait for the main menu to load again. But other than that it's all working surprisingly perfectly.

[cridenour]

Glad to hear it. I never needed to do that with ENet so that's where I hope switching to the sockets and having a clearer connection state to report to Godot will not make that necessary.