Server-spawned client replication

[philpax]

Hi again!

I was looking at the client replication example, and noticed that it supports two modes: client-spawned with replication to the server which can replicate it to everyone else, and prespawn in which the client spawns the entity and the server then takes over authority.

However, I'd like to spawn the entity on the server, and have a client replicate its version back to the server (i.e. the opposite of client prespawn). My use case is that I'm doing simulation on the client, but want the server to control spawning of all synchronised entities.

Is this possible? Additionally, can the client providing replication change? (e.g. if another client is better placed to simulate, I'd like to be able to make it take over and start providing sync)

[cBournhonesque]

Hi, I'm not sure I understand. So there's 2 separate things: 1) you want to spawn the entity on the server, but then transfer authority to the client? (i.e. client updates are replicated to the server) 2) you want to be able to transfer the 'owner' of an entity (i.e. the entity that is sending replication changes) to another client? so the receiver entity stays the same, but the sender entity is different since it belongs to a new client? Would the new sender entity already exist on the second client?

Do you have more context on why you need this? I don't get the "I'm doing simulation on the client, but want the server to control spawning of all synchronised entities." part. I'm surprised that the client has authority.

[philpax]

Yep, that's correct on both fronts! I recognise this is somewhat of an unusual scenario, but I have a large physics world in which the server only does light validation (I'm not too concerned about cheaters for my usecase). There are definitely drawbacks to this, but it's the best fit for my project. I can DM you more details if you'd like, but I'd like to validate that I can make the sync work here before talking about my specific project publicly. (I promise I'm not building an indie mega-project MMO destined to fail!)

I'd like to spawn the entities on the server so that the server is authoritative over the existence of entities, but the clients are authoritative over the state of the entities.

The clients simulate the entities they own and send their changes to the server, which will then replicate the changes to the other clients. When a client leaves the simulation range of an existing entity, or another client with a better simulation is available, the ownership is transferred to the other client, and they become responsible for simulation and replication. The second client would likely already have this entity present, unless in exceptional cases (which I'm happy to deal with as they occur).

I'm happy to implement all of the ownership transfer etc logic myself, but I'm not sure what the best way to handle the replication up to the server would be. If Lightyear's current support for client replication can't support this, that's fine - I can just manually send up the client state to the server via messages - but it'd be nice to take advantage of it if possible!

[cBournhonesque]

Hm, for 1) I would probably just:

• send a message to the server that client X wants to spawn a new entity; the server checks the validity of the message and sends a message back, upon which the client creates an entity and starts replicating. One thing to keep in mind in this case is that the client doesn't know the server's entity (only the receiver of the replication has the entity mapping from the remote world to the local world), which might be needed for 2)

2) For this, I think you could implement this like this:

• Send a message TransferOwnershipTo { new_client: ClientId, owner_entity: Entity, // previously owning entity on the client side receiver_entity: Option<Entity>, // recipient entity on the server side } to the server, and remove Replicate on the entity.
• in the server, receive the message, fill in the receiver_entity by using the internal entity mapping, and send the message to the new owning client. Maybe also remove the existing entity mapping?
• in the new owning client, add Replicate on the new owning entity.

I will add a field TargetEntity on the Replicate component. If TargetEntity == TargetEntity::Spawn, the remote peer will just spawn a new entity as normal. If TargetEntity == PreExisting(Entity) (where Entity is the remote entity, i.e. the server's entity in this case), then the receiver won't spawn a new entity, but will just update its internal entity mapping and keep going.

What do you think?

Thanks for creating the issue btw, it pushes me to think more deeply about authority/ownership of entities and how to make it more flexible! Ideally I would like a simple transfer_ownership command that just transfers the ownership of an entity either to the remote, or to another client.

[philpax]

Apologies for the late reply!

For 1), I think it would be the other way around? The server would say "I want this entity to be spawned", and then the client would then create it and replicate it upwards. However, that would still be a bit troublesome as the server would have a "ghost entity" that would be converted to a real entity by the replicating client.

For 2), that sounds promising! I see you've already implemented it; I'll have to play around with it. I agree that a simple transfer_ownership command would be ideal; as it is, I think most users of Lightyear don't have to worry about the distinction between the local- and the remote-entity, and what we've discussed so far might weaken that abstraction.

---

This is the workflow I'd like to eventually support:

• the server spawns some entities that clients can simulate; these live entirely on the server to begin with
• the clients are made aware of the entities (perhaps through rooms, or perhaps globally to begin with); the entities are inert to begin with, aside from what the server might be doing with them
• a client enters simulation distance of one or more of these entities, and the server assigns ownership to this client
• this client starts simulating the entity and replicating its new state to the server, which will then replicate it to the other clients
• eventually, the client leaves (the area, the server, etc), causing the server to reassign simulation/replication to another client in the area, with the new client continuing as-is from the last state it received from the server
• eventually, there are no clients capable of simulating the entity, ownership is reassigned to the server, and the entity goes inert again
• eventually, another client can appear and take ownership again, and the cycle repeats

Essentially, the server has the final say on who owns the entity (and it can choose to own the entity), and there may be a peer supplying replicated simulation state for that entity to the server, which will then replicate it back to the other clients.

The issues I see are:

• your current model might not allow for the server to do the ownership transfer; instead, the client might need to explicitly hand it off? Not sure on this

• the server might want to change the state (i.e. teleport the entity, restore its health, etc) without taking simulation away from the peer. I'm not sure how to solve this; there could be a temporary ownership transfer, but that would halt the simulation.

  In previous work, I've just sent a message telling the replicating peer what its new state for that entity should be - that might still be the best way to deal with it, I can't think of an alternative that wouldn't lead to inconsistent state somewhere.

Keen to hear your thoughts!

[cBournhonesque]

Ah I see, I was thinking that for 1) you wanted the clients to initiate the spawning of entities, and the server to validate that the request was valid/correct. But it looks like you actually want to spawn the entities on the server directly.

For your workflow, I think it's do-able. Let's take it step-by-step:

the server spawns some entities that clients can simulate; these live entirely on the server to begin with

So server spawns entity S.

the clients are made aware of the entities (perhaps through rooms, or perhaps globally to begin with); the entities are inert to begin with, aside from what the server might be doing with them

Let's say we use rooms. The server adds a component Replicate with replication_target = All and replication_mode = Room. Clients receive the replication updates and spawn an entity. Client 1 has the mapping: remote = S -> local = C1 Client 2 has the mapping: remote = S -> local = C2

a client 1 enters simulation distance of one or more of these entities, and the server assigns ownership to this client this client starts simulating the entity and replicating its new state to the server, which will then replicate it to the other clients

A client joins the room for one of the entity. The server sends a message TransferOwnership(S, C1) to C1. The server removes the Replicate component on S. Client 1 uses its internal entity_map to map from S to C1. Client 1 adds the Replicate component on C1 with target_entity: S to avoid spawning a new entity on the server.

The server needs to update the replication_target of S so that it excludes C1: A) currently I think updating the replication_target isn't supported, it would do nothing. B) the second option would be to remove the Replicate component on S, and then re-add it with the correct replication_target. I think that would actually work. On other clients, we wouldn't spawn a new entity because the clients already have a mapping S -> CN. If a mapping already exists, we don't spawn a new entity. C) or you could just make C1 leave the room, so that the client updates aren't sent to C1 anymore.

A) or C) are probably the cleanest solutions.

eventually, the client leaves (the area, the server, etc), causing the server to reassign simulation/replication to another client in the area, with the new client continuing as-is from the last state it received from the server

So the server would be the one detecting the ownership change? In that case the server would send TransferOwnership(S, C2) to both C1 and C2. C1 sees that the ownership is taken from them and removes the Replicate component. C2 sees that the ownership is given to them adds the Replicate component with the correct replication_target and target_entity = S. (we can also just have 2 separate messages GiveOwnership and RemoveOwnership)

The server needs to update the replication_target of S so that it excludes C2 and includes C1: A) and B) are unchanged C) we make C1 join the room, and C2 leave the room.

eventually, there are no clients capable of simulating the entity, ownership is reassigned to the server, and the entity goes inert again

The server detects that C2 cannot own the entity, but also there are no clients that can own the entity; we again update the Replicate component accordingly, and the entity is inert because the server doesn't update the entity.

---

I think some of the key insights are:

• the server doesn't have to do complicated target_entity shenanigans, it can just update the replication_target field, or simply update the rooms
• if an entity already exists in the receiver's mapping, adding the Replicate component again on the sender will not re-spawn the entity on the receiver (only relevant if you remove/re-add the Replicate component, but still good to know)
• there seems to be only one message needed, TransferOwnership, so I might be able to provide an abstraction for this whole flow. I just struggle to really understand the concepts of ownership/authority. Is it simply the peer who is sending replication updates for an entity?

[cBournhonesque]

Actually, I still don't really understand what you mean by "I want the client to take ownership of an entity". I get that you want the client to have the Replicate component so that it can send updates to the server.

But does that just mean that you want the client to be able to 'control' the entity? i.e. control its movements, etc. In that case it would be easier to just update the LeafwingInput component, which will cause the client to send input messages to the server; the server will still do the simulation and then rebroadcast the state to all clients.

Or is it that you actually want the client to have ownership to reduce the simulation load on the server?

[philpax]

Thanks for that analysis! I think that all makes sense - I'm not sure about the rooms, because I need to think about how rooms will interact with the large world that I have (i.e. I need to control entity stream distance, and that may be separate from simulation distance), but it's definitely a good starting point.

What happens if two clients end up replicating at the same time? i.e. the transition period where ownership is being transferred and both clients have Replicate on their side for a given entity. Does the server just ignore one? Does it apply data from whichever one sent data last?

Or is it that you actually want the client to have ownership to reduce the simulation load on the server?

That's correct. I'm going for a "thin server" model, where the clients contribute their simulation of their local world (or what they're assigned to), and the server validates and replicates the relevant state to the other clients. My project has unique constraints that make server-side simulation difficult, which is why I'm doing it this way instead of a much simpler server-authoritative model.

[cBournhonesque]

What happens if two clients end up replicating at the same time? i.e. the transition period where ownership is being transferred and both clients have Replicate on their side for a given entity. Does the server just ignore one? Does it apply data from whichever one sent data last?

I think it would apply data for the client who sent the initial Replication message last. That's a good point; you might need some coordination between the two clients to make sure that the previous one removes Replicate first. Maybe I could have a receiver work with two distinct sender entities, but that seems a bit premature/complicated for now.

[philpax]

I was thinking about the problem and realised that I already have an OwnerId component that's used by the server to create the Replicate component that replicates to the other clients. It occurs to me that this could also be used by the clients to create/destroy their own Replicate components, such that:

• a server creates an entity with no OwnerId (i.e. implicitly owned by the server) and a Replicate component that replicates to all clients
• a client comes across the entity
• the server creates an OwnerId component for that entity, assigning it to that client
• that client detects this through a system, and attaches its own Replicate component on its side to start replicating
• the server alters its Replicate for that entity to exclude that client, but still replicates its version of OwnerId (I don't think this is currently possible, the per_component_metadata's target is an intersection)
• at some point later, the server changes the value of OwnerId to another client
• the first client detects this and deletes its Replicate component
• the second client detects this and attaches its own Replicate component
• eventually, there are no simulating clients available, the OwnerId component is removed and the server takes full control again

I think the only thing blocking this would be the server-authoritative OwnerId; otherwise, it should be doable:

• a server system over Option<OwnerId> that alters the Replicate component to either replicate to all clients (if there is no OwnerId) or to all but the client marked by OwnerId
• a client system over Option<OwnerId> that creates the Replicate component if the owner ID exists and is equal to its local client ID, or removes it if the owner ID has changed/been removed

A security improvement would be to only allow replication updates for an entity from a specific client, so that other clients can't hijack the entity / the transferred-from client doesn't update the server's state during the switch. This is nice, but not necessary.

This would allow for a fully-declarative way of controlling who's replicating the entity, which should handle clients joining and leaving gracefully. Of note to #95 is that this would be a "user-space" concept; Lightyear still wouldn't know who the owner of an entity was. I'm not sure if that's a good or bad thing, I could see arguments for both ways.

[cBournhonesque]

Yes I think that would work, the OwnerId is basically an easy way to send TransferOwnership messages.

So the missing parts are:

• I can update the per_component_metadata target to be an override instead of an intersection, I think it might be clearer actually
• Being able to update the Replicate component at runtime. (I think removing/re-adding the component should work, but it would be better if it was just possible to update it)
• Be careful on the client-side to use the TargetEntity::PreExisting instead of TargetEntity::Spawn

A security improvement would be to only allow replication updates for an entity from a specific client, so that other clients can't hijack the entity / the transferred-from client doesn't update the server's state during the switch. This is nice, but not necessary.

Something like: "if the server receives updates from a client that is different from the target of the OwnerId component, ignore the update"?

[philpax]

Sounds good!

Something like: "if the server receives updates from a client that is different from the target of the OwnerId component, ignore the update"?

Yep; you might want a more general mechanism so that the server can validate / modify the update before replicating it out, but "ignore the update if it doesn't match" is sufficient for my use case.

[mmarklar]

But does that just mean that you want the client to be able to 'control' the entity? i.e. control its movements, etc. In that case it would be easier to just update the LeafwingInput component, which will cause the client to send input messages to the server; the server will still do the simulation and then rebroadcast the state to all clients.

Sorry to jump into an old thread but could you elaborate on updating the LeafwingInput component? I can't find that component in this or the leafwing repo. Can't find much about updating Bevy components in general.

My goal is to have the server assign "roles" to entities and then have the clients inputs change when their controlled entity is assigned a new role.

[cBournhonesque]

I think I meant the ActionState component. Would you mind asking in a new issue or in the discord? I'm not sure I understand your question

[cBournhonesque]

It is now possible to do this easily with those 2 PRs:

• https://github.com/cBournhonesque/lightyear/pull/594
• https://github.com/cBournhonesque/lightyear/pull/597

You can spawn an entity on the server, but then transfer authority to a client. The client will then simulate the entity and send replication updates for it.