Open ProfanedBane opened 4 years ago
Any steps to reproduce?
I can confirm this as well but god knows for repro steps.
I can confirm this as well but god knows for repro steps.
I can repro it now.
It's a different situation so not sure if it resolves the lag one.
That happens in almost every game (hold a key, unfocus window, come back and the key is jammed) is there really a way to fix that?
My repro for it is probably fine to ignore but people get it stuck during normal gameplay somehow.
Hmmm bumping this. I've noticed this happens when you have a poor connection, but considering this issue was opened 3 years ago it's probably a different issue. I've noticed that when I'm connected through TeamViewer to test something outside of my house, this happens a lot. Strangely enough, it can be resolved by raising the prediction tick bias in networking settings.
Actually, looking at the other comments, I think this is the same situation. It happens after the game would stutter.
The issue strikes hard when you're an engineer on an EVA mission, building something outside the station. Then JAM! you're running out of the station's grid into the void. And due to jetpacks being a limited resource, you can only toss all your belongings and scream into the radio asking for a help, trying to think of any reasonable IC explanation why have you attempted to suicide.
From what I've noticed so far. The jam seems to happen server-side and isolated to the InputMoverComponent.HeldMoveButtons
. Here is a screenshot with the issue reproduction on the 0237d095 build.
Notice that the client's debug overlay doesn't show any directional inputs pressed (the 'Walk' one is a red herring, it's lit because of the screenshot shortcut combination), while the server's HeldMoveButtons is set to Up.
I've got into the issue simply by running around the station using only the WASD keys to move my character. Here are the server's logs at the moment when the issue hit me.
[WARN] net.ent: Got late MsgEntity! Diff: -2, msgT: 58191, cT: 58193, player: localhost@JoeGenero
[WARN] net.ent: Got late MsgEntity! Diff: -2, msgT: 58191, cT: 58193, player: localhost@JoeGenero
[WARN] net.ent: Got late MsgEntity! Diff: -2, msgT: 58206, cT: 58208, player: localhost@JoeGenero
[WARN] eng: MainLoop: Cannot keep up!
[WARN] net.ent: Got late MsgEntity! Diff: -2, msgT: 58314, cT: 58316, player: localhost@JoeGenero
[DEBG] system.n_p_c: Sleeping mouse (39) (37112/n37112, MobMouse1)
[DEBG] admin.logs: Saving 4 admin logs.
[DEBG] system.n_p_c: Sleeping mouse (39) (37112/n37112, MobMouse1)
[DEBG] admin.logs: Saving 2 admin logs.
[WARN] eng: MainLoop: Cannot keep up!
[DEBG] admin.logs: Saving 2 admin logs.
[WARN] eng: MainLoop: Cannot keep up!
The reason why the jamming occurs is that the server can accidentally process input events out of order. If you take a closer look at the logs below, you may notice that the client's tick goes backward at some point. I'm not familiar with the game engine yet, but my best guess is that when the server's main loop can't keep up it may get behind clients, so there should be some cool mechanism to sync the server's current tick back to clients, which is when the bug with input jamming may happen.
I used ef4afc56 for testing and added logging to client-side and server-side input handlers.
_sawmillInputContext.Debug($"[{clientMsg.Tick}/{clientMsg.SubTick}/{ts}] f={clientMsg.InputFunctionId} state={clientMsg.State}");
[DEBG] input.context: [21737/ 4553/699067317883732] f=55 state=Down
[DEBG] input.context: [21768/ 8286/699067955264720] f=55 state=Up
[DEBG] input.context: [21778/31951/699068235025250] f=55 state=Down
[DEBG] input.context: [21793/38582/699068568776119] f=55 state=Up
[DEBG] input.context: [21806/44900/699068820163782] f=55 state=Down
[DEBG] input.context: [21798/ 8890/699068991730276] f=55 state=Up
Logger.Info($"[{_gameTiming.CurTick}/{subTick}/{ts}] {buttonsBefore}->{buttons} ({bit}, {enabled})");
[INFO] root: [21737/ 4553/699067525925274] None->Right (Right, True)
[INFO] root: [21768/ 8286/699068220934251] Right->None (Right, False)
[INFO] root: [21778/31951/699068416214053] None->Right (Right, True)
[INFO] root: [21793/38582/699068727281074] Right->None (Right, False)
[WARN] eng: MainLoop: Cannot keep up!
[WARN] net.ent: Got late MsgEntity! Diff: -7, msgT: 21798, cT: 21805, player: localhost@JoeGenero
[INFO] root: [21805/ 8890/699069207261654] None->None (Right, False)
[INFO] root: [21806/44900/699069228388281] None->Right (Right, True) <-- it gets jammed here
The large number I obtained via System.Diagnostics.Stopwatch.GetTimestamp(), which counts nanoseconds from some arbitrary starting time.
I noticed that FullInputCmdMessage
has a convenient InputSequence
field, which, I believe, can be used on the server-side to distinguish those pesky out-of-order key presses. Here is pseudocode-ish example:
HandleCmdMessage(IFullInputCmdMessage message) {
if (message.State == BoundKeyState.Down && lastObservedInputSequence > message.InputSequence) {
Logger.Warn("");
return false;
}
}
Here is me again with my daily findings. I've added more logging lines to the place where the client sends network messages and the place where the server receives the message. Interestingly, the jam isn't happening due to the network errors. It happens because the priority queue that sorts received messages by their ticks.
Here is the full story of how the bug reproduces:
I hope someone more experienced with the game engine can find the information helpful to create a solution to the issue because I don't believe I'll make on my own.
Nice find! It definitely feels like ticks should be monotonic; when the client rewinds to replay server messages, it should replay back up to the current tick instead of ending with a tick count lower than before. To that end, it may be necessary for the client to catch up over several frames and ensure that input is sent using the "real" current tick and not the replay-in-progress tick count.
Its been a long time since I last looked at input handling, but the client should send the inputs reliably while always incrementing InputSequence
, and the server shouldn't process them out of order (not saying that's what it does, but what it should do). It also needs to send input windows to make input more resilient to packet loss (e.g., see space-wizards/RobustToolbox/issues/3031). It shouldn't be hard to do, someone just needs to do it
would be nice if this got fixed, the 2 second lag anytime the server hits a lag spike is my nr.1 gripe.
i have found the culprit of my issue specifically. Vsync+linux= obvious extremely bad behavior.
In laggy situations your inputs can get stuck "on" and constantly move you in a direction until you mash the offending key to reset it.