Server disconnects from Bolt service during long scene load

celfox commented 5 years ago

How to reproduce issue

Start server.
Load large scene that takes several minutes to load.

Expected Behavior

Bolt should continue functioning as normal. Console should work, and should accept new connections.

Actual behavior

Bolt systems completely break. Console does not work, and it does not accept new connections. Output log states: (snipped to useful parts for relevance)

4/9/2019 11:08:36 PM: State changed to: Disconnected
4/9/2019 11:08:36 PM: Changing Connect State: Idle => DisconnectPending
4/9/2019 11:08:36 PM: Got a unexpected Disconnect in LoadBalancingClient State: Disconnected. Server: GameServer Trace:   at ExitGames.Client.Photon.LoadBalancing.LoadBalancingClient.OnStatusChanged (ExitGames.Client.Photon.StatusCode statusCode) [0x00000] in <6dba35f2952c4339affc4b678681e241>:0 
  at ExitGames.Client.Photon.PeerBase+<>c__DisplayClass146_0.<EnqueueStatusCallback>b__0 () [0x00000] in <fd2f6251eac24567aae6ddc195cd3a56>:0 
  at ExitGames.Client.Photon.EnetPeer.DispatchIncomingCommands () [0x00000] in <fd2f6251eac24567aae6ddc195cd3a56>:0 
  at ExitGames.Client.Photon.PhotonPeer.DispatchIncomingCommands () [0x00000] in <fd2f6251eac24567aae6ddc195cd3a56>:0 
  at ExitGames.Client.Photon.PhotonPeer.Service () [0x00000] in <fd2f6251eac24567aae6ddc195cd3a56>:0 
  at ExitGames.Client.Photon.LoadBalancing.LoadBalancingClient.Service () [0x00000] in <6dba35f2952c4339affc4b678681e241>:0 
  at udpkit.platform.photon.PhotonPoller.Update () [0x00000] in <6dba35f2952c4339affc4b678681e241>:0 

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 45)

4/9/2019 11:08:36 PM:   at udpkit.platform.photon.PhotonPoller.BoltDisconnect () [0x0000b] in <6dba35f2952c4339affc4b678681e241>:0 
  at udpkit.platform.photon.PhotonPoller.OnConnectStateChange (udpkit.platform.photon.ConnectState state) [0x00081] in <6dba35f2952c4339affc4b678681e241>:0 
  at udpkit.platform.photon.PhotonPoller.set_ConnectState (udpkit.platform.photon.ConnectState value) [0x00045] in <6dba35f2952c4339affc4b678681e241>:0 
  at udpkit.platform.photon.PhotonPoller.ChangeState (udpkit.platform.photon.ConnectState state) [0x00006] in <6dba35f2952c4339affc4b678681e241>:0 
  at udpkit.platform.photon.PhotonPoller.HandlerStateChange (ExitGames.Client.Photon.LoadBalancing.ClientState lbState) [0x0001c] in <6dba35f2952c4339affc4b678681e241>:0 
  at (wrapper delegate-invoke) System.Action`1[ExitGames.Client.Photon.LoadBalancing.ClientState].invoke_void_T(ExitGames.Client.Photon.LoadBalancing.ClientState)
  at ExitGames.Client.Photon.LoadBalancing.LoadBalancingClient.set_State (ExitGames.Client.Photon.LoadBalancing.ClientState value) [0x00015] in <6dba35f2952c4339affc4b678681e241>:0 
  at ExitGames.Client.Photon.LoadBalancing.LoadBalancingClient.OnStatusChanged (ExitGames.Client.Photon.StatusCode statusCode) [0x00347] in <6dba35f2952c4339affc4b678681e241>:0 
  at ExitGames.Client.Photon.PeerBase+<>c__DisplayClass146_0.<EnqueueStatusCallback>b__0 () [0x0000c] in <fd2f6251eac24567aae6ddc195cd3a56>:0 
  at ExitGames.Client.Photon.EnetPeer.DispatchIncomingCommands () [0x00042] in <fd2f6251eac24567aae6ddc195cd3a56>:0 
  at ExitGames.Client.Photon.PhotonPeer.DispatchIncomingCommands () [0x00034] in <fd2f6251eac24567aae6ddc195cd3a56>:0 
  at ExitGames.Client.Photon.PhotonPeer.Service () [0x00005] in <fd2f6251eac24567aae6ddc195cd3a56>:0 
  at ExitGames.Client.Photon.LoadBalancing.LoadBalancingClient.Service () [0x0000f] in <6dba35f2952c4339affc4b678681e241>:0 
  at udpkit.platform.photon.PhotonPoller.Update () [0x00019] in <6dba35f2952c4339affc4b678681e241>:0

Configuration

Unity Version: 2018.3.9f1
Bolt Version: 1.2.7
Operating System: Windows 10

ramonmelo commented 5 years ago

@celfox Have you tried with Bolt 1.2.8?

celfox commented 5 years ago

Sorry for the massive delay, updating to 1.2.8 broke a lot of things and it took a while to fix them.

After trying the update, I can confirm it does exactly the same thing.

ramonmelo commented 5 years ago

Hello @celfox,

After several tests and research, this is not directly related to Bolt, but on how you handle your new scene. When you call BoltNetwork.LoadScene, Bolt will start to load the Scene using the SceneManager.LoadSceneAsync, which is the recommended way to load heavy scenes.

The main load process occurs in the background as expected, and at this step, Bolt stills responsive. But after this, Unity needs to activate your scene, and at this moment it can freeze if you have several GameObjects to be initialized (call Awake() and Start()), and yes, at this moment, any other script may not run, including the ones from Bolt. We do not have control over this. Keep in mind that all loading occurs in the main thread.

In your large scene, the GOs are all active in the hierarchy and their components enabled? All rendered or in the camera's field? Are you using LODs? Maybe the trick is to activate/enable by batch or only what's needed, or even instantiate on demand by batch and pool, etc. You could also split the scene into sub-scenes and load them additively.

In order to mitigate any disconnections from the Photon Service, Bolt already maintains a background thread to send ack messages to our server, so your peer will not be disconnected for a while (about 1:30 minutes). We can't maintain this indefinitely because this also happens when your applications go to background, and it must be disconnected after some period of inactivity.

Here are some links with some discussion about this topic:

https://stackoverflow.com/questions/50618388/what-is-happening-in-scenemanager-loadsceneasync https://forum.unity.com/threads/scenemanager-loadsceneasync-freeze-in-2018.543316/ https://gamedev.stackexchange.com/questions/130180/smooth-loading-screen-between-scenes

BoltEngine / Bolt-Tracker