Nodes auto replication

[kochol]

I want to add a new feature to replicate the nodes from the server to clients so the clients can join in the middle of the game without any works.

This is my repo for this issue I'm currently working on https://github.com/kochol/godot/ in network-replication branch I will be happy to get reviews from you.

The way I will do this is by adding a new property to Node class to set replication.

[willnationsdev]

Don't know anything about the best way to implement this, but anything that simplifies the networking workflow / automates some of the process for users is a win in my book.

[Xrayez]

Supporting this feature, I should note that it would not only be restricted to networking, but also gives opportunity for implementing replay system that builds upon replication system as can be seen in Unreal Engine.

[kiiada]

This would be a great time saving feature. Right now it looks like some people are using hacked together approaches involving a secondary source of persistence providing scene state to the client on connect. It would be much simpler to have this functionality built in

[trooper]

Are there any updates on this one? I see that the branch from the original post was not updated since a year ago. Can this issue be broken down into subtasks, I'd be happy to try to help with some of these, but I'm just getting ramped up on the whole Godot engine, so it's probably too much to take the whole thing at once.

[Xrayez]

@trooper I was on a search on implementing proper multiplayer replication support since, and I've found some useful book/code related to this:

ReplicationManagerServer.cpp ReplicationManagerClient.cpp

So basically a node could have three replication actions in terms of networking:

• ACTION_CREATE, create node (spawn on server → spawn on client);
• ACTION_UPDATE, update node (if node exists on client);
• ACTION_DESTROY, destroy node (delete on server → delete on client).

All these actions need to ensure consistency. For instance, if for some reason a client managed to delete its node on it's own (cheating), the ReplicationManager on server should force to create such node on client again even if action is ACTION_UPDATE.

Networking should be efficient, so not all node properties need to be replicated (textures, materials etc). So it makes sense to implement some kind of property info for each property related to it's replication, like:

• Is it being replicated at all? is_replicated or...

• How often should it replicate/update?

  • REPLICATE_NEVER - should not be replicated;
  • REPLICATE_ONCE - only on node creation;
  • REPLICATE_STEP - update only at fixed step (every 3 frames, for instance), useful replicating and interpolating motion transform;
  • REPLICATE_ALWAYS - critical info that needs to arrive every frame

According to these replication update modes for each property, the replication manager could decide whether data should be sent reliably (TCP) or unreliably (UDP). Node creation/deletion should always be reliable, for instance.

All these concepts can be implemented using already powerful animation system, if you give it a thought (it even handles needed interpolation for networking):

So there could be some kind of Replication track that could handle node replication, but instead it would be recorded/processed internally:

As I mention about recording, there needs to be another subsystem in ReplicationManager that handles that. So it would "insert" specific keys for replicated properties inside internal ReplicationPlayer (similarly to animation player), and add those property keys to clients simultaneously to their own replication players that would re-create the same behavior as on the server.

That's why I was saying that it would be useful for implementing replay system, or in fact as a prerequisite for it, as it tightly related to replication. Replay system has an ability to record and playback recorded states. It just happens that the server is in "Record mode", while it pushes recorded properties to clients in "Playback mode". In fact I have some prototype/proof-of-concept implementation I fiddled around:

StateRecorder.gd

I think this feature requires quite a lot of engine refactoring/rewrite, so proper foundation by the core devs needs to be established for this, or at least give approval as to how to approach this properly. Hopefully I've set the direction at least!

[wbulot]

Precisely I'm looking for a way to accomplish node replication in the case of a player's reconnection and this feature would save a lot of time.

I don't know if anyone is working on it but I send them courage :).

[Catchawink]

@Xrayez, looking at the animation system is a great idea.

I built a replication system in Unity (used for networking, replays, and importing/exporting assets) and would like to see something similar in Godot. The source is new to me but I'd like to work on this.

A good portion of replication definitely depends on how Godot keeps track of property changes and object creation/destruction. The animation system sounds like a good lead. Another lead might be Godot's undo/redo system, as it's presumably dealing with similar things. I'm going to look through more of your findings @Xrayez. If anyone has more information on these topics please share.

[Xrayez]

@Catchawink yeah, all these concepts fit together nicely. I believe replication system doesn't have to be embedded directly inside animation system, despite it sharing common features.

Godot does have UndoRedo class for editor, but I think it's too heavy and would be more suitable for turn-based games with TCP connection rather than real-time simulation.

ReplicationManager could act as a singleton in Godot, having different strategies over what, when and how data is recorded and trasmitted to other peers over the network (this should first be done locally). Memory management is critical here as well because of the sheer amount of data that needs to be stored during the game session, which could vary anything from two minutes to two hours, so it needs to be compressed as well (delta compression?) 😃

The nodes themselves need to tell which properties they need to replicate and convey that to ReplicationManager. Ideally every node created should be registered inside ReplicationManager via node_added scene tree signals, as well as node_removed. The node data should never be deleted (or deleted when memory is needed) because you want to be able to replicate the game state at any point, fetching nodes from ReplicationDB or similar:

So the replication data itself could be stored as keys containing time and data per property. If the replay is going to be played backwards, the creation action becomes destruction, and destruction becomes creation (sorry for making you dizzy)!

This has an additional benefit in case of out-of-order packets (too early or too late) as it doesn't matter for the ReplicationManager in what order packets arrive because each key has associated time value. One could go even more advanced and have the server play the game backwards, and the clients will obey just fine. 🤣

[Xrayez]

Following Godot's naming classes that are meant to work remotely, there could be RemoteAnimationPlayer or simply RemotePlayer which would basically act like recorder on the server side and playback the states over the network on different client machines.

[Catchawink]

@Xrayez great ideas and the visualizations help! This direction makes a lot of sense.

One question I have is at what level clients would keep track of replication. I'll try to illustrate a scenario.

Say a client joins a game instance midway through. The server could synchronize the client's state to the current game state in three ways:

1. Sending the client a full history/replay of past events. This keeps the client totally in sync with the server. This would be ideal if the client wanted to easily replay any part of the game on demand, but would/could be a lot of info to send over the network at the moment a player joins the game. As you said @Xrayez, a game session could last for hours! Imagine hours of property changes sent over.

This approach works for short games but doesn't scale well to longer ones. Which leads to option 2...

2. Sending the client a snapshot of the current game state (as opposed to a history of property changes). A snapshot would essentially be a Godot scene file, but produced by the server mid-game. The server would basically save a scene file that represents the current game state and send that file to a client. This has the potential advantage of catching the client up to speed with the game state quicker than option 1, because the size of the data doesn't increase with the game length. One catch is that the client would have to request information about replays from the server at some later point.

This approach could be better than option 1 but has limitations when you're dealing with a large, dynamic map. Imagine a game that occurs on a custom forest level. Assume that the level's layout is easily subject to change through player interactions. Something nice about option 1 is that it can send clients info it's already serialized before--the initial forest map, followed by incremental changes to the map. When something changes, the server can simply add that change to a list/cache of other serialized changes.

Option 2 doesn't have the same benefit. It would involve the server serializing a new snapshot of the forest every time a client joins. That could get very expensive. In a game involving many players, the map would be re-serialized constantly. Which leads to option 3...

3. Sending the client a relatively recent snapshot of the game state, followed by a replay of game states occurring between that snapshot and the current game state. This is better--the server doesn't have to constantly produce snapshots and the data sent to a client is still pretty small.

The server could even calculate the optimal trade-off between snapshots-to-replays.

There's actually another option to consider, which is sending deltas between states instead of replays. Replays would contain all changes between two states, whereas a delta would contain the minimum differences. Replacing replays in this scenario with deltas would generally be better, I think, but there could still be a use case for replays if the performance cost of calculating deltas was too high.

Hopefully this made some sense. This kind of avenue of thought might seem excessive but I think an implementation of replication that takes all of these things into account (deltas, replays, snapshots) would not only scale well but be applicable beyond networking and replays. Deltas could be used in an asset versioning system, for instance.

[Xrayez]

@Catchawink good observations, what you described could be handled via different replication strategies as one alternative.

1. Sending the client a full history/replay of past events

As you said, this would be more suitable for short games and would simplify the task for the programmer, say the strategy would be called STRATEGY_SYNC_ALL.

2. Sending the client a snapshot of the current game state

This strategy would be called STRATEGY_SYNC_CURRENT.

It would involve the server serializing a new snapshot of the forest every time a client joins.

If that's a problem, the map could be cached, and for very large level would need to be stored on disk (?). That also depends on whether a forest is a custom made level (user-generated content) or built-in level within a project, so it would have to be serialized (mostly) every time because the only instance exist on player's machine. With many players (more than 8-16), that's the realm of MMOG's that such replication manager would choke on so these kind of use cases are not as general purpose I suppose. As I said before, the whole scene doesn't have to be sent, the programmer could decide what is static and what should be present within a project initially on all machines and have him responsible over what needs to be replicated.

3. Sending the client a relatively recent snapshot of the game state

I think that's where you're heading to what I'm going to propose. Sending only recent snapshot wouldn't guarantee the correct replication of states though. So what could be done instead is to differentiate what properties are considered critical to replication (see original comment):

enum ReplicateMode {
    REPLICATE_NEVER,
    REPLICATE_ONCE,
    REPLICATE_STEP,
        REPLICATE_CHANGED,
    REPLICATE_ALWAYS,
}

So the property replication track which is configured as REPLICATE_ONCE would only be replicated on ACTION_CREATE. So in the case of a forest level, incremental changes to the map would be configured as REPLICATE_CHANGED. If a client joins the server mid-game, it would fetch property replication tracks that are critical to recreate the forest, and any changes would be applied to initial state of that forest, irregardless of the keys positions on a replication's track timeline (this is how animation system works with the key set to UPDATE_CONTINUOUS rather than UPDATE_DISCRETE).

That means if forest is created on game start, if there are keys (properties) that are not critical to replication (update states of objects that are already destroyed), these could be tossed away safely without risking desynchronization issues, synchronization is async in terms of tracks:

There's actually another option to consider, which is sending deltas between states instead of replays.

If using deltas, that would again require that the full history of game states are in sync on clients joining the server. So this would be useful for optimizing network traffic in the long run but initial state would still be needed to be replicated.

Deltas could be used in an asset versioning system, for instance.

I've been looking for some implementations for delta compression and found this C implementation, which is used in Fossil SCM but as it claims it could be applied to different domains.

[Catchawink]

If that's a problem, the map could be cached

@Xrayez right. Caching makes sense if serialization causes an overhead. A server could recycle cached snapshots and deltas. This would definitely be advanced and would be far from an initial priority.

for very large level would need to be stored on disk (?). That also depends on whether a forest is a custom made level (user-generated content) or built-in level within a project, so it would have to be serialized (mostly) every time because the only instance exist on player's machine.

If levels are user-generated then that opens the door to replicating textures, meshes, and audio that isn't built-in on the client side. It would be ideal to store that type of information on the client's disk. I've managed a lot of this in Unity by writing some systems on top of the engine. It would be easier in Godot, I believe, having access to the source.

With many players (more than 8-16), that's the realm of MMOG's that such replication manager would choke on so these kind of use cases are not as general purpose I suppose.

This would definitely be more advanced and mostly applicable to MMO's, as you were suggesting.

As I said before, the whole scene doesn't have to be sent, the programmer could decide what is static and what should be present within a project initially on all machines and have him responsible over what needs to be replicated.

Definitely, if the scene has static elements then the entire thing doesn't have to be sent. In any case where an object or property is static, the system can and should optimize for that. And as you said, static properties and built-in assets are likely to be used in many projects.

So what could be done instead is to differentiate what properties are considered critical to replication

Optimizing at the level of properties makes sense. It seems like a good place to start.

If using deltas, that would again require that the full history of game states are in sync on clients joining the server. So this would be useful for optimizing network traffic in the long run but initial state would still be needed to be replicated.

Right, although the initial state could be a cached state on the server.

I've been looking for some implementations for delta compression and found this C implementation, which is used in Fossil SCM but as it claims it could be applied to different domains.

This seems like a good lead, but how about rolling with our own solution? A delta can be treated like one frame of animation that moves from one scene state to another. By building on top of that insight and the format of files in Godot we might optimize things better.

[Xrayez]

This seems like a good lead, but how about rolling with our own solution?

Yeah it just takes some design work. It's just that when it comes to serialization it made sense to me that the data will most likely be represented in binary form. Actually I've played with these concepts already and gave it more attention towards how recorded states could be efficiently saved on disk as replays. The basic datatypes in Godot could take a lot of space when done via GDScript with var2bytes or similar (actual structures took me more data than states data itself in some cases), yet it was definitely just a proof-of-concept so this could be improved on the source level I think.

[SaracenOne]

I actually have my own networking solution for this sort of thing. It's undocumented and is probably broken, but it does exactly what I believe people here are asking about. If there's enough interest, I could probably spare some time to give it some more attention (https://github.com/SaracenOne/network_manager)

[Xrayez]

@SaracenOne cool, this is pretty much reflects what I have in mind in terms of CREATE/UPDATE/DESTROY replication actions, yet it seems to neglect storing replication data to be played back locally. The idea with custom networking reader/writer is also nice. And it seems like it's only limited to syncing entities' transform for now.

[kiiada]

Storing replication actions is a neat idea, but it seems to be a distinct and separate feature. For instance, storing actions to play back locally is something that some single player game designs could benefit from, and many multiplayer games would prefer to opt out of. A 60min+ FPS match or a semi-persistent world in the vein of a minecraft server would accumulate huge amounts of replay data.

Replication of a scene state should ideally send as little data as possible, which would mean sending either the full scene once for the client to replicate or a delta between the client and server scene.

[Xrayez]

For instance, storing actions to play back locally is something that some single player game designs could benefit from, and many multiplayer games would prefer to opt out of.

I feel like my described approach would be more general-purpose, and with enough knobs it could suit many use cases, both single player and multiplayer. In your example, it would just mean not storing states, but rather act like synchronization system, either at initial state and/or during gameplay itself (or even on-demand synchronization).

[Razzeeyy]

Hello! I've wrote an addon for networked object spawning/despawning and also opened a similar issue at the new proposal repository.

https://github.com/godotengine/godot-proposals/issues/75

[DoubleDeez]

If anyone is looking for a solution now and is using C#, I just added automatic replication to the framework I'm building for my own game: https://github.com/DoubleDeez/MDFramework#automatic-member-replication

It's pretty basic, it just iterates your list of replicated members and checks if they've changed.

[Calinou]

Closing in favor of https://github.com/godotengine/godot-proposals/issues/75, as feature proposals are now tracked on the Godot proposals repository.