search

Unity-Technologies / com.unity.netcode.gameobjects

Netcode for GameObjects is a high-level netcode SDK that provides networking capabilities to GameObject/MonoBehaviour workflows within Unity and sits on top of underlying transport layer.
MIT License
2.15k stars 435 forks source link

NetworkVariable OnValueChanged trigger order on Client does not match Server/Host. #2122

Closed ezoray closed 2 years ago

ezoray commented 2 years ago

This is a longstanding issue that's been bugging me for a while which needs some clarification and hopefully a fix.

Description

Having a network object with a number of network variable fields, when multiple fields are changed on the server the order in which OnValueChanged events are triggered is correct on the server/host but out of order on the client. This can lead to awkward workarounds where game logic triggered by an OnValueChanged event relies on multiple fields of the object which are not yet available.

Reproduce Steps

Create a network object with multiple network variables, bools, ints, Vector2Ints for example, and change the values of these fields. On the server the OnValueChanged events occur in the order in which they are changed, on the client the order is not the same nor predictable.

For example, create a network object with the following fields, and change their values in the same order. IntegerOne IntegerTwo IntegerThree VectorOne VectorTwo VectorThree BoolOne BoolTwo BoolThree

Actual Outcome

Server/host OnValueChanged events order: IntegerOne IntegerTwo IntegerThree VectorOne VectorTwo VectorThree BoolOne BoolTwo BoolThree

Client OnValueChanged events order: BoolOne BoolThree BoolTwo IntegerOne IntegerThree IntegerTwo VectorOne VectorThree VectorTwo

Expected Outcome

Client OnValueChanged events should be triggered in the same order in which the variables are changed and match the OnValueChanged order of the server/host.

Environment

Additional Context

The results are from a test project which I can provide if necessary. In my game project with more going on the event trigger order can't be predicted.

jeffreyrainy commented 2 years ago

Hi @ezoray ,

That is a good observation. This subject can be a bit tricky (and lengthy), so I'll try to organize this reply in small sections. :-)

Hard facts: The current release of Netcode for GameObjects is designed to bundle all the updates to NetworkVariables done in the same frame together. There is no guarantee offered for the order in which they will be delivered, if modified on the same frame. There is no guarantee either about the frame number on which you'll get an updated value. NetworkVariables are meant for state, not as events.

Extra details: In fact, there's two concepts of frame, here. The frame on which the NetworkVariables were modified and the frame on which they were received. The deciding frame is on the send side. Everything that got modified in a given frame will be serialized at the end of the frame, in some order. This order decides delivery order on the receiving side. I would need to check, but because we failed (repeatedly) at including a snapshot system in this first release, I think there isn't even a guarantee that the client will receive all the updates in the same frame.

Rationale: We wanted a snapshot model. Everything that gets written in a given frame gets delivered together. This would be a good model as it gives the game devs some information. It also ease a lot of the processing: every new spawns happens first, everything gets updated, all the RPC are delivered, all the new despawns happen, etc... It can reduce the needed bandwidth as everything can be nicely packed.

But this is tricky when you consider that the bandwidth is limited. Then you have to pick a delivery model (most often eventual consistency) and there's plenty of little details we didn't get to implement. Interest management, for example. Hence, the snapshot system did not ship.

Work-around: If you need sequential updates done on the same frame to be delivered in a specific order, you need to use another mechanism. A few mechanisms come to mind:

Take-Aways: I'll discuss with the rest of the team about this, see if we can improve anything. Please consider the work-arounds above and let me know if you can find a way to address your issue.

ezoray commented 2 years ago

Hi Jeffrey, many thanks for the detailed explanation. That makes what's going on under the hood a lot clearer and why ordering the network variables isn't currently feasible.

I'm using OnValueChanged all the time and they do ultimately lead to state changes in either drawing or of the interface. As this is working fine in most situations I'll have a look at the logic for the couple of situations I have where more than one value are relied upon.

As I've tried to keep things consistent I've opted for just about everything from server to client going in the form of network variable changes. I have a dedicated object for handling RPC messages, mostly instigated by the client although there are some for synchronisation (scene management is disabled).

I'm making a fair amount of use of network lists, where previously I had a list of references for spawned objects I've done away with the need for them and replaced with a list of structs to do the same thing. They're a little more tedious to use but should mean less overhead than objects (in theory) and do allow the client to receive multiple values at once. Incidentally there's currently a bug with network lists which I've alluded to in #2105 (changes made to the list directly after object spawn are not triggering OnListChanged event).

I can see the benefits of changes reaching the client in order but if this isn't usually typical in multiplayer solutions then it is what it is and I'll change my code accordingly. Thanks for making me aware of it. :)

akoolenbourke commented 2 years ago

Hi @ezoray ,

That is a good observation. This subject can be a bit tricky (and lengthy), so I'll try to organize this reply in small sections. :-)

Hard facts: The current release of Netcode for GameObjects is designed to bundle all the updates to NetworkVariables done in the same frame together. There is no guarantee offered for the order in which they will be delivered, if modified on the same frame. There is no guarantee either about the frame number on which you'll get an updated value. NetworkVariables are meant for state, not as events.

Extra details: In fact, there's two concepts of frame, here. The frame on which the NetworkVariables were modified and the frame on which they were received. The deciding frame is on the send side. Everything that got modified in a given frame will be serialized at the end of the frame, in some order. This order decides delivery order on the receiving side. I would need to check, but because we failed (repeatedly) at including a snapshot system in this first release, I think there isn't even a guarantee that the client will receive all the updates in the same frame.

Rationale: We wanted a snapshot model. Everything that gets written in a given frame gets delivered together. This would be a good model as it gives the game devs some information. It also ease a lot of the processing: every new spawns happens first, everything gets updated, all the RPC are delivered, all the new despawns happen, etc... It can reduce the needed bandwidth as everything can be nicely packed.

But this is tricky when you consider that the bandwidth is limited. Then you have to pick a delivery model (most often eventual consistency) and there's plenty of little details we didn't get to implement. Interest management, for example. Hence, the snapshot system did not ship.

Work-around: If you need sequential updates done on the same frame to be delivered in a specific order, you need to use another mechanism. A few mechanisms come to mind:

Take-Aways: I'll discuss with the rest of the team about this, see if we can improve anything. Please consider the work-arounds above and let me know if you can find a way to address your issue.

That is very interesting to know. Can you tell me if there's any order gaurantee if you do a NetworkTransform/Rigidbody changes followed by an RPC? That is, will the client get the NetworkTransform/rb change processed before the RPC so the RPC can be assured the RB is up to date? I need to teleport on the server but a client-only RB that is following that server RB needs to know when to teleport and then go back to MovePosition updates. We want to avoid teleporting the client side object, going into MovePosition mode only to find the RB update hasn't come through and we MovePosition to the non-final position of the RB.

NoelStephensUnity commented 2 years ago

Hi akoolenbourke, We have a pending PR that fixes a bunch of issues with NetworkTransform. When this is merged you could wait for the patch or use the develop branch to get the fixes immediately. Based on your question, I think you would find this PlayerMovement.cs file useful to handle disabling the owner authoritative's RigidBody while teleporting.

As a side note, the NetworkTransform teleport has a bug when interpolation is enabled in the v1.0.0 package that causes the non-authority instances to look like they are fighting the position update with the RigidBody when in reality the interpolation is not being reset properly. As long as you have a NetworkRigidBody attached, all non-authoritative instances are automatically set to kinematic and the authority is the only instance that has kinematic disabled.

The up-coming patch should resolve the teleporting issue and if you use the same approach as the linked PlayerMovement.cs it will "just work".

akoolenbourke commented 2 years ago

Hi Noel. I might be not fully understanding the PlayerMovement script but it looks like it's client authoritative, but still exists and controls somethng on the server. Is that right? What we have is a situation where there exists a GameObject that's ONLY on the client's not on the server and it's an RB so it can collide with client side objects where it doesn't matter if each client has a different view of things etc. Things like debris hit by a vehicle that reacts realistically but only on that client, which is fine.

That object needs to follow (non-kinematically with MovePosition to get physics) a server authoritative object (the player) so it's positioned correctly but when that server authoritative object wants to teleport, we want to also teleport that client-only object. What we wanted to try and ensure is the situation where once we tell the client object to teleport that it can guarantee that the position it will teleport to is the position of the server authoritative object and to do that the transform of that SA object needs to be guaranteed to already be at the client. That is, if we teleported the RB on the server, then the server sent an RPC to the client to say "OK, teleport to X,Y,Z as well and then continue MovePosition from now on" that when tracking that server RB started up again (Client started MovePosition again) then the RB transform is already at the client for it to use.

I'm not sure that script solved that problem but I may not understand it properly

Thanks

NoelStephensUnity commented 2 years ago

@akoolenbourke In the PlayerMovement.cs file you will see this:

    /// <summary>
    /// Make this PlayerMovement-NetworkTransform component
    /// Owner Authoritative
    /// </summary>
    protected override bool OnIsServerAuthoritative()
    {
        return false;
    }

That will place the NetworkTransform into true owner authoritative mode for the updated version of NetworkTransform (in the develop branch today btw).

If you notice the rest of the code either exits early if it isn't the owner or executes certain blocks of code if it is the owner.

With the current version of NetworkTransform (i.e. v1.0.0), the owner authority wasn't really "true" owner authority but was "Server Authoritative Owner Directed"...which basically means the owner was allowed to send RPCs to the server and the server would update.

The updated version of NetworkTransform (this coming patch), when placed into owner authoritative mode the owner will immediately see the update to the transform, the server will receive the updated transform information when the current network tick the change occurred in is over, and then the server forwards those changes immediately to the clients (basically under the hood owner authoritative is a NetworkVariable with everyone read rights and owner write rights).

The fixes for the updated version of NetworkTransform resolve the issues with teleporting no matter what authoritative state you have chosen and with interpolation enabled. 👍

Does this help?

akoolenbourke commented 2 years ago

Yeah, can't wait to get the new NetworkTransform.

The issue I see with that script is that this object I am talking about exists ONLY on the client, there is no NetworkTransform. It's a client side only thing for client (That specific client) physics. It needs, at some point to be able to be 100% confident that some other object (that is networked) has has its position updated by the server so it can set its own position to that position then start following it again with MovePosition. This avoids a situation where it thinks it's allowed to follow again but the position hasn't been updated across the network yet so called MovePosition and doesn't teleport, going through other objects etc and reacting to them. Hope that makes sense.

jeffreyrainy commented 2 years ago

Closing the issue as we have a model where the order is not guaranteed.