GameObject.NetworkSpawn() lags the game and speeds up animations in a multiplayer setting

[yuberee]

Describe the bug

In a multiplayer setting with multiple players connected, running NetworkSpawn() makes all the animations speed up like crazy and the game starts lagging.

I could narrow the problem down at this line: https://github.com/Small-Fish-Dev/Sauna/blob/871867ef08a73f9d7d170130b413f011b3c4affe/code/Game/NpcSpawnArea.cs#L78

To Reproduce

(As seen in the video)

1. Clone https://github.com/Small-Fish-Dev/Sauna/tree/main
2. Start a game in editor mode
3. Join locally (Or with any other player)
4. Run in console "EndSession" and click the Continue button (Or StartSession in console)

Expected behavior

.

Media/Files

Video:

https://www.youtube.com/watch?v=_F8t5VLqJWQ

Additional context

At one point we also received an animation error I've never seen before, it's probably not the cause but worth looking at:

2024/03/14 01:53:45.8341    [GameMenu] SceneAnimationSystem.UpdateAnimation: One or more errors occurred. (Collection was modified; enumeration operation may not execute.)    System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
   at Sandbox.ComponentList.GetAll[T](FindMode find)+MoveNext()
   at Sandbox.SceneAnimationSystem.UpdateInThread(SkinnedModelRenderer renderer)
   at Sandbox.SceneAnimationSystem.<UpdateAnimation>b__1_0(SkinnedModelRenderer x)
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass44_0`2.<PartitionerForEachWorker>b__1(IEnumerator& partitionState, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()

Nothing interesting in the logs. Oddly enough it's not NetworkSpawn in general but it's just the NetworkSpawn we use to spawn NPCs, but we couldn't trace it back.

[garrynewman]

I couldn't really see any point in the video where the animations speed up like crazy. Does the performance graph show anything different?

[yuberee]

Here's how it looks without lagging:

https://github.com/Facepunch/sbox-issues/assets/59583743/e4958e38-db89-48f7-ae80-37e94b7d242b

And here it is live with an extra crash added in:

https://github.com/Facepunch/sbox-issues/assets/59583743/2b50f4f3-8ffc-4770-a317-fc926130fbe6

We were also calling NetworkSpawn() on every player accidentally but even after making sure the entire logic gets run on the host only the same thing happens.

Let me get the performance graph in a second

[yuberee]

Here is the performance graph, nothing too weird but the speed up only happens on the host now.

https://github.com/Facepunch/sbox-issues/assets/59583743/46da9614-a54e-4457-b29d-bfd37924974b

[kurozael]

Pretty sure I fixed this - was related to EditLog. Can you confirm @yuberee?