Torch freezes sometimes when restart is meant to happen.

[XeroCreator]

• [I am not sure...] Verify that this issue is related to Torch and not a Torch plugin or the vanilla game
• [not really, but keeps happening] Ensure that the issue is reproducible for testing (provide a link to a test world if necessary)
• [no] Is this a suggestion?

Torch Version: v1.3.1.33-Patron, SE Version: 1.190.101

What's happening to me and others , is the torch DS hangs sometimes on restart. I do 6 hour restarts and it happens frequently enough that I have to check status in discord to make sure it's starting and stopping correctly. It doesn't actually crash, it just freezes (for ex Uptime: 06:00:18 was the last reading) I can't use gui, can't close from the X, I have to force close it from task manager each time.

attached all logs available but there was no dmp file. torch-issue.zip

[Grovespaz]

I have the same issue, this is what the log shows for me, which is the same for XeroCreator's logs:

23:54:06.4151 [INFO]   Keen: Automatic stop after 2 hours and 0 minutes
23:54:06.4326 [INFO]   Keen: Exiting..
23:54:06.4326 [INFO]   TorchSessionManager: Unloading torch session for Alasdairia
23:54:06.4326 [INFO]   Keen: TORCH MOD: Unregistering mod communication.
23:54:06.4424 [INFO]   MultiplayerManagerDedicated: Removed steam authentication intercept
23:54:06.4424 [INFO]   Keen: Used namespaces in scripts:
23:54:06.4424 [INFO]   Keen: Autosave in unload
23:54:06.4912 [INFO]   Keen: TORCH MOD: Exception occurred in communication thread! System.InvalidOperationException: The collection argument is empty and has been marked as complete with regards to additions.
   at System.Collections.Concurrent.BlockingCollection`1.Take()
   at Torch.Mod.ModCommunication.DoProcessing() in C:\jenkins\workspace\Torch_Torch_master\Torch.Mod\ModCommunication.cs:line 62
23:54:06.4912 [INFO]   Keen: TORCH MOD: COMMUNICATION THREAD: EXIT SIGNAL RECEIVED!
23:54:28.8995 [INFO]   Keen: Logging off Steam...
23:54:28.8995 [INFO]   Keen: Shutting down server...
23:54:29.1563 [INFO]   Keen: Done
23:54:31.3057 [INFO]   TorchSessionManager: Unloaded torch session for Alasdairia
23:54:31.3057 [INFO]   Keen: TORCH MOD: Unregistering mod communication.

Torch 1.3.1.49-master, SE 1.191.23

[N1Ran]

Check your SpaceEngieers-Dedicated.cfg. Session is ending after set times. It's basically doing what it's set to do base off that log. Look for AutoRestartEnabled>true</AutoRestartEnabled>

ps. this is not a Torch related question/issue evidence by "Keen: Automatic stop after 2 hours and 0 minutes"

Xerocreator, your issue may be fixed by using Ale's watchdog. It tracks restarts and force closes the window whenever it hangs to continue the restart process. https://torchapi.net/plugins/item/370c6d2e-c2db-4eb2-a04f-49c9ee89dd2b

[XeroCreator]

I have the same issue, this is what the log shows for me, which is the same for XeroCreator's logs:

23:54:06.4151 [INFO]   Keen: Automatic stop after 2 hours and 0 minutes
23:54:06.4326 [INFO]   Keen: Exiting..
23:54:06.4326 [INFO]   TorchSessionManager: Unloading torch session for Alasdairia
23:54:06.4326 [INFO]   Keen: TORCH MOD: Unregistering mod communication.
23:54:06.4424 [INFO]   MultiplayerManagerDedicated: Removed steam authentication intercept
23:54:06.4424 [INFO]   Keen: Used namespaces in scripts:
23:54:06.4424 [INFO]   Keen: Autosave in unload
23:54:06.4912 [INFO]   Keen: TORCH MOD: Exception occurred in communication thread! System.InvalidOperationException: The collection argument is empty and has been marked as complete with regards to additions.
   at System.Collections.Concurrent.BlockingCollection`1.Take()
   at Torch.Mod.ModCommunication.DoProcessing() in C:\jenkins\workspace\Torch_Torch_master\Torch.Mod\ModCommunication.cs:line 62
23:54:06.4912 [INFO]   Keen: TORCH MOD: COMMUNICATION THREAD: EXIT SIGNAL RECEIVED!
23:54:28.8995 [INFO]   Keen: Logging off Steam...
23:54:28.8995 [INFO]   Keen: Shutting down server...
23:54:29.1563 [INFO]   Keen: Done
23:54:31.3057 [INFO]   TorchSessionManager: Unloaded torch session for Alasdairia
23:54:31.3057 [INFO]   Keen: TORCH MOD: Unregistering mod communication.

Torch 1.3.1.49-master, SE 1.191.23

The keen restart is not enabled on my server, I actually remember talking to jimmacle about that and it him saying it might be better to just disable it anyway via torch.

However the other section about the communication thread seems to be the issue.

I actually see this in other logs where restarts were successful without the freezing.

`20:43:42.3274 [DEBUG] Keen: Session snapshot save - END 20:43:42.3541 [INFO] Keen: Exiting.. 20:43:42.3541 [INFO] Keen: TORCH MOD: Unregistering mod communication. 20:43:42.3541 [INFO] Keen: TORCH MOD: Exception occurred in communication thread! System.InvalidOperationException: The collection argument is empty and has been marked as complete with regards to additions. at System.Collections.Concurrent.BlockingCollection1.Take() at Torch.Mod.ModCommunication.DoProcessing() in C:\jenkins\workspace\Torch_Torch_Patron\Torch.Mod\ModCommunication.cs:line 62 20:43:42.3541 [INFO] Keen: TORCH MOD: COMMUNICATION THREAD: EXIT SIGNAL RECEIVED! 20:43:42.3698 [INFO] Keen: Used namespaces in scripts: 20:43:42.3698 [DEBUG] Keen: MySession::Unload START 20:43:42.3698 [DEBUG] Keen: AutoSaveInMinutes: 10 20:43:42.3698 [DEBUG] Keen: MySandboxGame.IsDedicated: True 20:43:42.3698 [DEBUG] Keen: IsServer: True 20:43:42.3698 [INFO] Keen: Autosave in unload 20:43:42.3698 [DEBUG] Keen: Saving world - START 20:43:42.3698 [DEBUG] Keen: Making world state snapshot. 20:43:42.3698 [DEBUG] Keen: Before snapshot.: GC Memory: 3,938,147,048 B 20:43:42.8698 [DEBUG] Keen: After snapshot.: GC Memory: 3,947,824,592 B 20:43:42.8698 [DEBUG] Keen: Directory cleanup: GC Memory: 3,947,865,552 B 20:43:42.8698 [DEBUG] Keen: Saving world - END 20:43:42.8698 [DEBUG] Keen: Session snapshot save - START 20:43:42.8698 [DEBUG] Keen: Checking file access for files in target dir. 20:43:49.7612 [DEBUG] Keen: Session snapshot save - END 20:43:49.7924 [DEBUG] Keen: Unloading Procedural World Generator 20:43:55.4024 [DEBUG] Keen: Network readers disposed 20:43:55.6524 [DEBUG] Keen: Network readers disposed 20:43:55.6524 [INFO] Keen: Logging off Steam... 20:43:55.6524 [INFO] Keen: Shutting down server... 20:43:55.7618 [INFO] Keen: Done 20:43:55.7618 [DEBUG] Keen: MyDefinitionManager.PreloadDefinitions() - START 20:43:55.7618 [DEBUG] Keen: MyDefinitionManager.PreloadDefinitions() - END 20:43:57.4651 [DEBUG] Keen: MySession::Unload END `

It appears that the server saves, unloads, and stops fine... but this is 100% the log from when it froze. It doesn't happen often enough to frequently check logs so it's hard to track down.

[Grovespaz]

@N1Ran I ported my server over from a Vanilla server. The automatic restart works fine on the vanilla server, it's only on Torch that it doesn't come back online. Autorestart is enabled: <AutoRestartEnabled>True</AutoRestartEnabled>

This leads me to think it is a Torch issue, unless Torch doesn't support the automatic restart?

[N1Ran]

Torch does not play well with Vanilla's Auto Restart. Use AutoCommands for that

[Profit0004]

I see this in the log when it fails usually.

22:40:00.3418 [INFO] Keen: Automatic stop after 4 hours and 0 minutes 22:40:00.3589 [INFO] Keen: Exiting.. 22:40:00.3629 [INFO] Keen: TORCH MOD: Unregistering mod communication. 22:40:00.3629 [INFO] Keen: TORCH MOD: Exception occurred in communication thread! System.InvalidOperationException: The collection argument is empty and has been marked as complete with regards to additions. at System.Collections.Concurrent.BlockingCollection`1.Take() at Torch.Mod.ModCommunication.DoProcessing() in C:\jenkins\workspace\Torch_Torch_master\Torch.Mod\ModCommunication.cs:line 62 22:40:00.3629 [INFO] Keen: TORCH MOD: COMMUNICATION THREAD: EXIT SIGNAL RECEIVED!

[XeroCreator]

I see this in the log when it fails usually.

22:40:00.3418 [INFO] Keen: Automatic stop after 4 hours and 0 minutes 22:40:00.3589 [INFO] Keen: Exiting.. 22:40:00.3629 [INFO] Keen: TORCH MOD: Unregistering mod communication. 22:40:00.3629 [INFO] Keen: TORCH MOD: Exception occurred in communication thread! System.InvalidOperationException: The collection argument is empty and has been marked as complete with regards to additions. at System.Collections.Concurrent.BlockingCollection`1.Take() at Torch.Mod.ModCommunication.DoProcessing() in C:\jenkins\workspace\Torch_Torch_master\Torch.Mod\ModCommunication.cs:line 62 22:40:00.3629 [INFO] Keen: TORCH MOD: COMMUNICATION THREAD: EXIT SIGNAL RECEIVED!

N1Ran has already said this is caused by keen's auto restart. Disable it and it should go away.

If not, make your own issue, this is not directly related to my issue.

[rexxar-tc]

Exceptions about torch mod are irrelevant. It's a warning and doesn't affect anything.