Messages seem to be processed using mainthread, causing freezes and connection problems

[LordTylus]

Hello there,

today with discord being unavailable it came to my attention, that our servers kept crashing when people used the public chat. I used vv1.7.0

Also we noticed especially when 30+ people are on our server constantly talking (even when discord is up) that stuff

It appears there is a lot of stuff happening on mainthread which seem to cause connection issues and can pause the entire server if networking is taking a bit longer.

18:54:10.8764 [INFO]   Chat: [Global:0] Zuckerstift: but this streaming bug is cancer
18:54:30.8859 [ERROR]  SEDiscordBridge.SEDiscordBridgePlugin: SendChatMessage: One or more errors occurred.
18:54:30.9537 [INFO]   Chat: [Global:0] Sidonus: crash inc
18:54:51.0879 [ERROR]  SEDiscordBridge.SEDiscordBridgePlugin: SendChatMessage: One or more errors occurred.
18:54:51.1477 [INFO]   Chat: [Global:0] Sidonus: here we go again
18:55:11.3996 [ERROR]  SEDiscordBridge.SEDiscordBridgePlugin: SendChatMessage: One or more errors occurred.
18:55:11.4255 [INFO]   Chat: [Global:0] Lighthammer: yes
18:55:28.1209 [ERROR]  SEDiscordBridge.SEDiscordBridgePlugin: SendChatMessage: One or more errors occurred.
18:55:28.1428 [INFO]   Chat: [Global:0] firzen1: if the server does not get fixed within the next hour, we will need to postpone the race
18:55:30.4596 [ERROR]  Torch: Frozen thread dump Main thread
Common prefix:
   at System.Threading.ManualResetEventSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.SpinThenBlockingWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.InternalWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at SEDiscordBridge.DiscordBridge.SendChatMessage(String user, String msg)
   at SEDiscordBridge.SEDiscordBridgePlugin.MessageRecieved(TorchChatMessage msg, Boolean& consumed)
   at Torch.API.Managers.MessageRecievedDel.Invoke(TorchChatMessage msg, Boolean& consumed)
   at Torch.Managers.ChatManager.ChatManagerClient.RaiseMessageRecieved(TorchChatMessage msg) in C:\jenkins\workspace\Torch_Torch_master\Torch\Managers\ChatManager\ChatManagerClient.cs:line 156
   at Torch.Managers.ChatManager.ChatManagerClient.Multiplayer_ChatMessageReceived(UInt64 steamUserId, String messageText, ChatChannel channel, Int64 targetId, String customAuthorName) in C:\jenkins\workspace\Torch_Torch_master\Torch\Managers\ChatManager\ChatManagerClient.cs:line 139
   at System.DelegateExtensions.InvokeIfNotNull[T1,T2,T3,T4,T5](Action`5 handler, T1 arg1, T2 arg2, T3 arg3, T4 arg4, T5 arg5)
   at Sandbox.Engine.Multiplayer.MyMultiplayerBase.RaiseChatMessageReceived(UInt64 steamUserID, String messageText, ChatChannel channel, Int64 targetId, String customAuthorName)
   at Sandbox.Engine.Multiplayer.MyDedicatedServer.OnChatMessage(ChatMsg& msg)
   at Patched_Sandbox.Engine.Multiplayer.MyMultiplayerBaseOnChatMessageReceived_Server_0(ChatMsg )
   at Sandbox.Engine.Multiplayer.MyMultiplayerBase.OnChatMessageReceived_Server<>VRage_Network_ChatMsg.Invoke(IMyEventOwner& <static>, ChatMsg& msg, DBNull& arg2, DBNull& arg3, DBNull& arg4, DBNull& arg5, DBNull& arg6)
   at VRage.Network.CallSite`7.Invoke(BitStream stream, Object obj, Boolean validate)
   at VRage.Network.MyReplicationLayer.Invoke(CallSite callSite, BitStream stream, Object obj, EndpointId source, MyClientStateBase clientState, Boolean validate)
   at VRage.Network.MyReplicationServer.OnEvent(MyPacketDataBitStreamBase data, CallSite site, Object obj, IMyNetObject sendAs, Nullable`1 position, EndpointId source)
   at VRage.Network.MyReplicationLayer.OnEvent(MyPacketDataBitStreamBase data, NetworkId networkId, NetworkId blockedNetId, UInt32 eventId, EndpointId sender, Nullable`1 position)
   at VRage.Network.MyReplicationLayer.ProcessEvent(MyPacketDataBitStreamBase data, EndpointId sender)
   at VRage.Network.MyReplicationLayer.OnEvent(MyPacket packet)
   at Sandbox.Engine.Multiplayer.MyTransportLayer.ProcessMessage(MyPacket p)
   at Sandbox.Engine.Multiplayer.MyTransportLayer.HandleMessage(MyPacket p)
   at Sandbox.Engine.Networking.MyReceiveQueue.Process(NetworkMessageDelegate handler)
   at Sandbox.Engine.Networking.MyNetworkReader.Process()
   at Sandbox.MySandboxGame.Update()
   at Sandbox.Engine.Platform.Game.UpdateInternal()
   at Patched_Sandbox.Engine.Platform.GameRunSingleFrame_0(Object )
   at Sandbox.Engine.Platform.FixedLoop.<>c__DisplayClass11_0.<Run>b__0()
   at Sandbox.Engine.Platform.GenericLoop.Run(VoidAction tickCallback)
   at Sandbox.Engine.Platform.Game.RunLoop()
   at Sandbox.MySandboxGame.Run(Boolean customRenderLoop, Action disposeSplashScreen)
   at Torch.VRageGame.DoStart() in C:\jenkins\workspace\Torch_Torch_master\Torch\VRageGame.cs:line 245
   at Torch.VRageGame.Run() in C:\jenkins\workspace\Torch_Torch_master\Torch\VRageGame.cs:line 118
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Suffix 0
   at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj)

Suffix 1
   at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj)

Suffix 2
   at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj)

18:55:30.4626 [FATAL]  Initializer: System.TimeoutException: Server watchdog detected that the server was frozen for at least 60 seconds.
   at Torch.Server.TorchServer.CheckServerResponding(Object state) in C:\jenkins\workspace\Torch_Torch_master\Torch.Server\TorchServer.cs:line 253
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()
System.TimeoutException: Server watchdog detected that the server was frozen for at least 60 seconds.
   at Torch.Server.TorchServer.CheckServerResponding(Object state) in C:\jenkins\workspace\Torch_Torch_master\Torch.Server\TorchServer.cs:line 253
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()```

[FabioZumbi12]

For who had this problem, test this version: SEDiscordBridge.zip

[LordTylus]

Sadly this version did not fix it.

We still encounter various connection problems when many messages are sent to the game.

Also notice simspeed dropping from 1.0 to 0.8 whenever 1 message is sent to chat.

according to profiler my basegame needs 12ms to update. To get to a 0.8 simspeed 1 update should need 20.75ms (16,6/0.8) so 20.75-12 = 8.75ms are occupied from the server whenever a message is sent. (of course this number is not constant)

when 2 players send messages at the same time the server encounters hard laggs and when unlucky freezes completely.

When I turn SEDiscordBridge off entirely. all problems vanish, no simspeed decrease, no connection problems nothing like that.

Would be amazing if you could take an other look at it as it is an amzing plugin and helps us a lot administrating the servers. But for the time being I had to disable it. Because the current advantanges dont outweigh the performance drawbacks.

[FabioZumbi12]

Ok, just tryied the Jimmacle recommendations but semms not working too. Try this version: SEDiscordBridge.zip

[LordTylus]

This version seems to work fine.

I dont experience any hard simspeed drops or connection problems yet

Using the version from last night it sinks to 0.8 Using the version from 2 hours ago it stays on 1

So far it looks to be working pretty pretty good