giantotter / giantotter_public

2 stars 0 forks source link

Mutliplayer handshaking not always working #545

Closed dwenger01 closed 8 years ago

dwenger01 commented 8 years ago

1) A little background - I have some local code changes that support reading in spawn pose data from the project file (dave_rest_spawn in this case), and it also reorders the roles so the customer shows up first. So when the customer spawns in the code detects that he should be sitting in chair.01. This causes him to try to sit, and even though I have him spawning in very close to the chair he thinks he needs to perform a GOTO operation. Everything works correctly in "single player" - i.e. no other players join.

2) In the case when the 2nd player joins, everything in the first player's world is deleted. All player one's scene objects go away, the loading screen is displayed again, and then the cue card finally comes up with the timer reset after the scene reloads. The 2nd player (waitress) has loaded up by this point and its cue card is timing down.

3) Eventually the first player rejoins, but at this point he is not visible to the waitress. i.e. Player 1 (customer) can see Player 2 (waitress), but not vice versa.

I've set breakpoints in the room_bot code and each player is definitely joining the room as I'd expect. They also receive their OnMsgNewPlayer() messages in the Character class - however it seems that the customer (who I have the debugger attached to) is not told about NewPlayer for the waitress. He gets the chef and and bartender, but no waitress NewPlayer() message. It just seems like there is a race condition where Player 1 is not available to receive the waitress NewPlayer() since he has not yet loaded up himself when she tells him she's joined.

After the first or 2nd game play when the scenario ends and new one begins both players seem to get all the messages and see each other correctly. The issue just seems exacerbated in the initial load case.

I'm still a bit confused as to why Player 1 completely blows away his scene and resets when Player 2 initially joins, but perhaps this is the easiest way to reset player 1's scene state. I would think he would just hang out indefinitely without the other player until Player 2 joins, then just reset the initial world state (from cache perhaps) and allow Player 2 to join his scene.

The other thing I'm slightly confused about is that the Character class is the class that handles the OnMsgNewPlayer() message. I think in the case where we're not seeing the waitress join Player 1's game it's because there is no Character instance available to handle the message. Is there any way that the message could be handled by a different class? Or maybe I'm misunderstanding what's happening here, but it looks like the Customer Character instance is the one handling when the chef and bartender enter the scene, but when the customer is not there when the waitress joins, we are not able to add the waitress to the scene.

jefforkin commented 8 years ago

Dave, question for clarification:

Do you start client 1 (customer) and immediately start client 2 (waitress)? Or, do you wait for the customer to start the game (past the cue card), and then start client 2?

Do you see different behavior in both of these cases, or do they both exhibit the bug?

I just tried both of these possibilities, with the code you sent me for sit changes, and actually did not see the bug in either case. So, it definitely seems to be a race condition, exacerbated by your hardware and/or your geographic distance from the IRC server.

I can poke around a bit more and try to force the bug to occur, and alternatively I can look for ways to fortify the code to try to eliminate the issue, even if I cant repro it -- like possibly adding some kind of acknowledgement when a player joins a room (or some related event occurs), so that perhaps the roombot could resend critical messages.

dwenger01 commented 8 years ago

Hi Jeff.

1) I start up both clients to the screen that says "Tutorial" and "Multiplayer". 2) I join multiplayer with client 1 (the client to which my debugger is attached). 3) After client 1 gets to its cue card I join multiplayer with client 2 4) Client 1 resets shortly thereafter and I'm seeing client 2 arrive at its cue card while client 1 is still on its loading screen. 5) Client 1 loads and reaches its cue card 6) Both clients' cue cards count down to 0 and both clients show the scene. 7) At this point client 1 does not see the waitress, but client 2 sees both players.

Note also that client 1 is a debug build and client 2 is a release build, so client 2 tends to connect and load much faster than client 1.

jefforkin commented 8 years ago

I still have no repro'd the issue, but have one more question:

The game starts and client 1 does not see the waitress. If the waitress moves on client 2, does she then appear on client 1? Or she still never appears?

The reason I am asking is that, since you said that chat is working, they seem to be in the room together, which means they should receive ObjectUpdate messages when someone moves. And, the OnObjectUpdate() function in NetManager has code to spawn a character when it receives an update from someone who does not yet exist.

You mentioned that client 1 never receives an OnNewPlayer message for the waitress. But all that message does is tell the customer that someone new joined, so he should set his dirty flag, which then causes NetManager.BroadcastObjects() to broadcast a current state update to everyone. But the same thing will happen anyway the next time a player moves or rotates.

But if client 1 never sees the waitress, even after she walks around, then it seems like he's not receiving any ObjectUpdates for her, which is odd. Let me know if this is the case and I can try to diagnose how that could ever happen -- receiving chat but not receiving ObjectUpdates....

dwenger01 commented 8 years ago

I was actually seeing both cases occur - i.e. in some runs client 1 would not see client 2 and vice versa in other runs.

To answer your question, in the case where the customer (client1) can't see the waitress I observed that the waitress object had never been created in client1's scene. So even if the waitress moved client1 could not see her. It seemed that the message for creating a new waitress character was never received -or- handled by the client1 instance.

The opposite would occur as well (where client2 could not see the customer) at times, but by and large I would see the case where the customer could not see the waitress.

jefforkin commented 8 years ago

But that's what I find weird....

In NetManager.OnObjectUpdate(), there is code that says, if I receive a message from a Player that has never been created, then spawn her.

So, what you are seeing implies that the Waitress is not sending object updates. This could be true if for some strange reason she never receives a OnJoinedRoom message, so she never sets her matchmakingState to InRoom, in which case NetManager.BroadcastObjects() will not send any updates for her.

I can look at the roombot code to try to see how that could ever happen.

But if that did happen, we would see the behavior you are describing -- chat works, but client 1 never sees client 2's avatar.

dwenger01 commented 8 years ago

Right. Yeah, maybe that's been the disconnect for me all along. I haven't run with a debugger on both clients, so I haven't verified whether or not the waitress got her joined room message. But I would think that if she didn't get the message then she would never actually think she'd joined the room and would never load up the scene for that client.

When I was running room_bot in the debugger I'm pretty sure I was seeing both players' request to join a room being handled by the bot. I just never verified that the waitress client received the join message from the bot.

jefforkin commented 8 years ago

Actually, I've proven my last theory wrong. If the waitress does not receive the OnJoinedRoom message, then she never actually makes it beyond the loading screen. So that is not what's happening.

I would be curious to know if when you run Client 1 through the debugger, when the waitress does not appear for Client 1, if you then set a breakpoint in NetManager.OnObjectUpdate(), do you ever hit that breakpoint?

Players only send object updates when the dirty flag is set. Other than a few places where we set the dirty flag when the level first loads, we only set that flag for head rotation -- in NetCharacter.SyncHeadLook(). If for some reason, the Waitress never thinks she turns her head at all, she'll never send an update. So, if for some reason the slow level loading causes the Customer to miss the initial updates, you might never get a later update.

One thing you could try is setting the dirty flag more often -- like whenever a player walks around, or whenever chat gets sent. I'd be curious to know if that helps, but that's just a band aid, not a complete fix.

Though I can't repro what you are seeing in TRG, I have seen something similar when running 3 clients for a education GroupPlay. The first player could see the other player, but the second player and spectator could not see the first player. I am going to switch gears and test with an education project, because it seems like it may be the same issue.

dwenger01 commented 8 years ago

That sounds like a good approach - the test via education project.

I will test the other case in my repro scenario. i.e. See if NetMAnager.OnObjectUpdate() is hit on client 1 when the waitress is changing her head rotation.

jefforkin commented 8 years ago

I am able to easily repro the issue (I think its the same issue) with an education project, and catch it in the debugger. What I'm seeing is that for some reason, for one client NetManager.BroadcastingObjectUpdates is set to false. So that client never calls BroadcastObjects(), regardless of the Dirty flag.

Haven't yet tracked down how this occurs, but should be able to figure it out soon. Will update when I've got it.

dwenger01 commented 8 years ago

Cool. Sounds related, if not the same thing I'm seeing. Thanks Jeff.

dwenger01 commented 8 years ago

Verified that when the waitress player rotates that the customer's client gets the message. I'm having trouble repro-ing the customer not having the waitress character instantiated in the scene (i.e. she's always being created now), but the waitress client is now not seeing the customer player even if he moves around and rotates. So I think perhaps your theory about the client not removing the room's reference to the character that was created and then deleted might be correct. That seems to explain why messages from the unseen player would not cause a new player to be created.

jefforkin commented 8 years ago

I believe this is fixed with patch 8eec171. Dave please give it a try.

The issue that I was seeing occurred when Client2 sent an ObjectUpdate while Client1 was in the middle of loading the level. Client1 would spawn Client2's character, and then the level loading code would destroy all the objects, but Client2's character's NetObject remained in NetManager's NetworkedObjects list, as a degenerate destroyed object. Once the NetworkedObjects list was corrupted, this could cause a failure to start the UpdateObjects coroutine, and Client1 would never send any object updates to anyone.

I did two things to fix this: 1) NetManager will not spawn a character in response to an ObjectUpdate if ScenarioManager is in the middle of a level load. 2) NetObjects now have an OnDestroy() that tries to remove themselves from the NetManager's NetworkedObjects list.

dwenger01 commented 8 years ago

Hi Jeff. Your patch fixed the issue. Nice job!