LCD Panel related crashes

[midspace]

Zenduin on Discord gave me this one:

Exception occured: System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
   at Sandbox.ModAPI.MyTerminalControlFactoryHelper.GetProperty(String id, Type blockType)
   at Sandbox.ModAPI.Interfaces.TerminalPropertyExtensions.GetValue[T](IMyTerminalBlock block, String propertyId)
   at Economy.scripts.Management.TextPanelWriter.Clear()
   at Economy.scripts.Management.TextPanelWriter.Create(IMyTextPanel textPanel)
   at Economy.scripts.Management.LcdManager.ProcessLcdBlock(IMyTextPanel textPanel)
   at Economy.scripts.Management.LcdManager.UpdateLcds()
   at Economy.scripts.EconomyScript.<Timer1EventsOnElapsed>b47_0()
   at Sandbox.MySandboxGame.ProcessInvoke()
   at Sandbox.MySandboxGame.Update()
   at Sandbox.Engine.Platform.Game.UpdateInternal()
   at Sandbox.Engine.Platform.Game.RunSingleFrame()
   at Sandbox.Engine.Platform.FixedLoop.<>cDisplayClass1.<Run>b__0()
   at Sandbox.Engine.Platform.GenericLoop.Run(VoidAction tickCallback)
   at Sandbox.Engine.Platform.Game.RunLoop()
   at Sandbox.MySandboxGame.Run(Boolean customRenderLoop, Action disposeSplashScreen)
   at SpaceEngineers.MyProgram.Main(String[] args)

This is another explicit call to : FontSize = ((Sandbox.ModAPI.IMyTextPanel)_panel).GetValueFloat("FontSize");

I've asked Zenduin for the full log file, as I'm going to log this with Keen. We shouldn't be seeing these kind of exceptions.

[jpcsupplies]

So the catch was unable to nail it. I've not seen the issue on my server, and it has literally dozens of LCDs - so presumably it is vageuly related to SESE?

[midspace]

It's a different location to before, but the same API as before.

[midspace]

Attached the log file, which appears to be from SESE. SEServerExtenderLog-2017-02-14.txt

[jpcsupplies]

Complete speculation: Looks to me like the wrapper (reflection?) is crashing when modapi is looking for an object that probably has been unloaded by sese to improve sim.

Best guess interpretation: On the first wrapper crash our lcd manager is looking for an LCD that was missing.. it all goes to crap and restarts server.

On the second crash a conveyor or refinery or similar factory block is trying to get the next ore in the conveyor network and crashing, followed by our LCD crashing becuase it cant find its LCD too The second crash also warns that a constructor was unloaded incorrectly (probably due to crash) and the render loop fails because it is trying to draw an object that is missing too.. then thre is a bunch of unintelligible sundry crashes related to assemblies (path overrides? exception on resource, exception enumerating next rare etc?) Then a thread timeout after it tries to unload two plugins..

SeSE then reboots DS, which runs for about 5 minutes, when the wrapper crashes a 3rd time - With another targetinvocationexception (missing object again?) Resource exception (missing ore or refinery??) Lcd manager update lcds (missing lcd?) Then all the same unsafe invoke get properties get value etc errors again (missing targets?)

4th crash seems similer to 3rd get value on a block that no longer exists etc

5th likewise, 6th likewise, 7th, 8th, 9th etc collection modified, operation may not execute..

10th gets intersting again.. at 2017-02-14 06:04:51.7630 object reference not set to an instance of an object, Updatebeforesimulation etc.. sounds once again like its trying to access something that got removed - in this error it appears Economy wasn't running at all (perhaps admin disabled it? there is a few chat override notifications suggesting a conversation beween admin and players ) but other than the missing economy trap, we get all the other same errors.

11th crash, economy is back again -and collection was modified error again. 12th, 13th, 14th

15th (2017-02-14 08:29:45.1801) again they appeared to try running without economy - and we see another object reference not set to an instance error again followed by a pile of wait handle errors

16th, 17th, 18th, 19th, 21st, 22nd, economy is back again collection modified error

23rd at 2017-02-14 15:23:30.2933 they again try to launch without economy and we see object reference error again

It then runs 7 hours, then at 2017-02-14 22:58:07.1614 the 24th crash with Key not found error - error returning control thrown by logging?? this crash looks particularly nasty. Then it looks to sit there half dead doing saves for a while and thats the end of the log.

[midspace]

Zenduin confirms to me that this has only started occurring after Keen added entity parallelization.

[midspace]

I haven't any more reports of this issue. So either we've trapped them all, or Keen have fixed them all.

[jpcsupplies]

good work