search

Neos-Metaverse / NeosPublic

A public issue/wiki only repository for the NeosVR project
193 stars 9 forks source link

Headless Unresponsive - System.ArgumentException: Invalid Raycast / System.InvalidOperationException / NaNsplosion #2914

Open Enverex opened 3 years ago

Enverex commented 3 years ago

Describe the bug?

No-one was able to access my headless as they'd receive a "Session ended" message. Trying to access the headless console for Neos resulted in no response from the headless console, it was effectively soft-frozen. The logs last entries referred to a NaNsplosion.

Relevant issues

Nothing definite. https://github.com/Neos-Metaverse/NeosPublic/issues/2855 could be related but that's based only due to the error listed.

To Reproduce

Unsure, I've not seen this before but it took down everything.

Expected behavior

For the headless to not completely and silently die.

Log Files

Here's the final snippet before everything died. The full log can be DMed over Discord or whereever on demand. It contains a lot of user details and asset IDs so avoiding posting it here publicly.

System.ArgumentException: Invalid Raycast. Origin: [-21.68194; 1.014251; -3.33854], Direction: [-Infinity; Infinity; Infinity], MaxDistance: 6.738106E-21
  at FrooxEngine.PhysicsManager.RaycastOne (BaseX.float3& origin, BaseX.float3& direction, System.Single maxDistance, System.Predicate`1[T] filter, System.Boolean hitTriggers, System.Nullable`1[T] debugDuration) [0x00031] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Physics.Raycaster.OnEvaluate () [0x000a9] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.LogixNode.CheckEvaluate () [0x00050] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Output`1[T].get_Value () [0x00008] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Output`1[T].get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Cast.CastClass`2[I,O].get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.WorldModel.GetSlot.get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Operators.MultiNullCoalesce`1[T].get_Content () [0x00015] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.WorldModel.GetTag.get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.String.Contains.get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Operators.ConditionalOperator`1[T].get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Operators.MultiNullCoalesce`1[T].get_Content () [0x00015] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.WorldModel.GetTag.get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.String.Contains.get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Operators.ConditionalOperator`1[T].get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].Evaluate (T def) [0x00010] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.DriverNode`1[T].FrooxEngine.LogiX.IDriverNode.EvaluateInputAndCache () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.LogixController.UpdateDriverNodes () [0x0009f] in <060518213ffe4796aa5ace79ea061f94>:0

   at void BaseX.UniLog.Error(string message, bool stackTrace)
   at void FrooxEngine.LogiX.LogixController.UpdateDriverNodes()
   at void FrooxEngine.LogiX.LogixController.Update()
   at bool FrooxEngine.World.RefreshStep()
   at bool FrooxEngine.World.Refresh()
   at void FrooxEngine.WorldManager.UpdateStep(double maxMilliseconds)
   at bool FrooxEngine.WorldManager.RunUpdateLoop(double maxMilliseconds)
   at void FrooxEngine.Engine.UpdateStep(double maxMilliseconds)
   at void FrooxEngine.Engine.RunUpdateLoop(double maxMilliseconds)
   at void NeosHeadless.Program.UpdateLoop()
   at void System.Threading.ThreadHelper.ThreadStart_Context(object state) in /home/ben/.cache/yay/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:line 71
   at void System.Threading.ThreadHelper.ThreadStart() in /home/ben/.cache/yay/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:line 109
11:51:51 PM.802 Exception when running synchronous action (Stage: Destructions):
System.ArgumentException: Invalid Raycast. Origin: [-21.68179; 1.014281; -3.338474], Direction: [-Infinity; Infinity; Infinity], MaxDistance: 4.492071E-21
  at FrooxEngine.PhysicsManager.RaycastOne (BaseX.float3& origin, BaseX.float3& direction, System.Single maxDistance, System.Predicate`1[T] filter, System.Boolean hitTriggers, System.Nullable`1[T] debugDuration) [0x00031] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Physics.Raycaster.OnEvaluate () [0x000a9] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.LogixNode.CheckEvaluate () [0x00050] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Output`1[T].get_Value () [0x00008] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Output`1[T].get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Cast.CastClass`2[I,O].get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.WorldModel.GetSlot.get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Operators.MultiNullCoalesce`1[T].get_Content () [0x00015] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.WorldModel.GetTag.get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.String.Contains.get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Operators.ConditionalOperator`1[T].get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Operators.MultiNullCoalesce`1[T].get_Content () [0x00015] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.RelayNode`1[T].get_Content () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].EvaluateRaw (T def) [0x0005c] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.Input`1[T].Evaluate (T def) [0x00010] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.LogiX.ProgramFlow.FireOnChangeBase`1[T].UpdateState () [0x00008] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.World.RunSynchronousActions () [0x0007b] in <060518213ffe4796aa5ace79ea061f94>:0
11:51:51 PM.814 Evaluate depth is above zero on LogiX Update
11:51:52 PM.892 Exception in background job:
System.InvalidOperationException: The change in tree cost is an invalid value, strongly implying the tree bounds have been corrupted by infinites or NaNs. If this happened in the broad phase's use of the tree, it's likely that there are invalid poses or velocities in the simulation, possibly as a result of bugged input state or constraint configuration. Try ru>
  at BepuPhysics.Trees.Tree.GetRefineTuning (System.Int32 frameIndex, System.Int32 refinementCandidatesCount, System.Single refineAggressivenessScale, System.Single costChange, System.Int32& targetRefinementCount, System.Int32& refinementPeriod, System.Int32& refinementOffset) [0x00012] in <f1c97308329249378dafd71e68e14cb9>:0
  at BepuPhysics.Trees.Tree.RefitAndRefine (BepuUtilities.Memory.BufferPool pool, System.Int32 frameIndex, System.Single refineAggressivenessScale, System.Single cacheOptimizeAggressivenessScale) [0x0002b] in <f1c97308329249378dafd71e68e14cb9>:0
  at FrooxEngine.BoundingBoxTree.RefitAndRefine () [0x00000] in <060518213ffe4796aa5ace79ea061f94>:0
  at FrooxEngine.DynamicBoneChainManager.UpdateAccelerationStructures () [0x00040] in <060518213ffe4796aa5ace79ea061f94>:0
  at BaseX.WorkProcessor.ProcessQueue (BaseX.WorkProcessor+ThreadWorker worker, System.Collections.Concurrent.ConcurrentQueue`1[T] queue) [0x00009] in <baa4c39c483a46c5907f6292aa763694>:0
11:52:58 PM.115 Peer Disconnected: [scrubbed]:64903, reason: Timeout, socketErrorCode: 0
1:35:30 AM.602  SignalR Reconnecting: System.TimeoutException: Server timeout (30000.00ms) elapsed without receiving a message from the server.
1:35:32 AM.378  SignalR Reconnected: siQ51KOuLpA5PYXRS5_quw3d676f6d1

Screenshots

No response

How often does it happen?

Sometimes

Does the bug persist after restarting Neos?

Yes

Neos Version Number

2021.9.1.710

What Platforms does this occur on?

Linux

Link to Reproduction Item/World

No response

Did this work before?

Yes

If it worked before, on which build?

No response

Additional context

I'm not sure if this was caused by something in the world or someone's avatar as it's not something I've experienced before.

Reporters

No response

Frooxius commented 3 years ago

Hmm from the log snippet it just looks like potentially faulty LogiX, but it's hard to say anything without context. It shouldn't cause crash by itself though, raycasts do not modify the simulation data. I'd need to see the whole log.

Does this happen in a particular world? Anything that can be reproduced on a regular client?

Enverex commented 3 years ago

I can't say what triggered it as I wasn't there at the time, I doubt the person that triggered it realised they did either. The worlds the server was hosting hadn't been changed so unless it's something incredibly obscure, I don't believe it would have been one of the worlds itself (otherwise I would expect it to have happened more often).

I reported this mainly due to the severity (complete hardlock of the Neos headless software leaving it in a zombie state) despite how unfortunately vague the report itself is, just in case something useful could be gleaned from the logs.

Logs sent as requested.

Frooxius commented 3 years ago

Looks like a bunch of invalid values made it to the acceleration structures used by the new dynamic bone collision query system. I've added guards in 2021.9.2.676 to prevent this which should fix it. Thanks for report! Let me know if the issue persists.

Enverex commented 3 years ago

Not sure if this is the same thing but the resulting behaviour was the same, the headless became unresponsive.

4:41:14 PM.043  Exception running asynchronous task:
System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object) ---> System.NullReferenceException: Object reference not set to an instance of an object
  at BepuUtilities.Collections.QuickSort.Sort[TKey,TComparer] (TKey& keys, System.Int32 l, System.Int32 r, TComparer& comparer) <0x4368bfd0 + 0x003b9> in <9b84f04af68846b0bf7215c07b3361e5>:0 
  at BepuPhysics.Trees.Tree.FindPartitionForAxis (BepuUtilities.BoundingBox* boundingBoxes, BepuUtilities.BoundingBox* aMerged, System.Single* centroids, System.Int32* indexMap, System.Int32 count, System.Int32& splitIndex, System.Single& cost, BepuUtilities.BoundingBox& a, BepuUtilities.BoundingBox& b, System.Int32& leafCountA, System.Int32& leafCountB) <0x4368b9f0 + 0x00072> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.FindPartition (BepuPhysics.Trees.Tree+SweepResources& leaves, System.Int32 start, System.Int32 count, System.Int32& splitIndex, BepuUtilities.BoundingBox& a, BepuUtilities.BoundingBox& b, System.Int32& leafCountA, System.Int32& leafCountB) <0x4368b340 + 0x00194> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.SplitLeavesIntoChildren (BepuPhysics.Trees.Tree+SweepResources& leaves, System.Int32 start, System.Int32 count, System.Int32 nodeIndex) <0x4368af80 + 0x000a6> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.CreateSweepBuilderNode (System.Int32 parentIndex, System.Int32 indexInParent, BepuPhysics.Trees.Tree+SweepResources& leaves, System.Int32 start, System.Int32 count) <0x4368acc0 + 0x00075> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.SweepBuild (BepuUtilities.Memory.BufferPool pool, BepuUtilities.Memory.Buffer`1[T] leafBounds) <0x43689ca0 + 0x0057b> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Collidables.Mesh..ctor (BepuUtilities.Memory.Buffer`1[T] triangles, System.Numerics.Vector3& scale, BepuUtilities.Memory.BufferPool pool) <0x43689730 + 0x0029c> in <f1c97308329249378dafd71e68e14cb9>:0 
  at CodeX.BepuMeshHelper.ComputeMeshCollider (BaseX.MeshX meshx, System.Boolean dualSided, BepuUtilities.Memory.BufferPool bufferPool, System.Object bufferLock, System.Action meshDataExtracted, BepuPhysics.Collidables.Mesh& mesh) <0x436817c0 + 0x0013d> in <368a2654c3714372b779a48d890211a7>:0 
  at FrooxEngine.MeshColliderData.BuildData () [0x000c1] in <d79cbb467f0147628a85804f9bdaa366>:0 
  at FrooxEngine.MeshPhysicsData`1[D].<EnsureUpToDate>b__30_0 () [0x000f3] in <d79cbb467f0147628a85804f9bdaa366>:0 
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object
  at BepuUtilities.Collections.QuickSort.Sort[TKey,TComparer] (TKey& keys, System.Int32 l, System.Int32 r, TComparer& comparer) <0x4368bfd0 + 0x003b9> in <9b84f04af68846b0bf7215c07b3361e5>:0 
  at BepuPhysics.Trees.Tree.FindPartitionForAxis (BepuUtilities.BoundingBox* boundingBoxes, BepuUtilities.BoundingBox* aMerged, System.Single* centroids, System.Int32* indexMap, System.Int32 count, System.Int32& splitIndex, System.Single& cost, BepuUtilities.BoundingBox& a, BepuUtilities.BoundingBox& b, System.Int32& leafCountA, System.Int32& leafCountB) <0x4368b9f0 + 0x00072> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.FindPartition (BepuPhysics.Trees.Tree+SweepResources& leaves, System.Int32 start, System.Int32 count, System.Int32& splitIndex, BepuUtilities.BoundingBox& a, BepuUtilities.BoundingBox& b, System.Int32& leafCountA, System.Int32& leafCountB) <0x4368b340 + 0x00194> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.SplitLeavesIntoChildren (BepuPhysics.Trees.Tree+SweepResources& leaves, System.Int32 start, System.Int32 count, System.Int32 nodeIndex) <0x4368af80 + 0x000a6> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.CreateSweepBuilderNode (System.Int32 parentIndex, System.Int32 indexInParent, BepuPhysics.Trees.Tree+SweepResources& leaves, System.Int32 start, System.Int32 count) <0x4368acc0 + 0x00075> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.SweepBuild (BepuUtilities.Memory.BufferPool pool, BepuUtilities.Memory.Buffer`1[T] leafBounds) <0x43689ca0 + 0x0057b> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Collidables.Mesh..ctor (BepuUtilities.Memory.Buffer`1[T] triangles, System.Numerics.Vector3& scale, BepuUtilities.Memory.BufferPool pool) <0x43689730 + 0x0029c> in <f1c97308329249378dafd71e68e14cb9>:0 
  at CodeX.BepuMeshHelper.ComputeMeshCollider (BaseX.MeshX meshx, System.Boolean dualSided, BepuUtilities.Memory.BufferPool bufferPool, System.Object bufferLock, System.Action meshDataExtracted, BepuPhysics.Collidables.Mesh& mesh) <0x436817c0 + 0x0013d> in <368a2654c3714372b779a48d890211a7>:0 
  at FrooxEngine.MeshColliderData.BuildData () [0x000c1] in <d79cbb467f0147628a85804f9bdaa366>:0 
  at FrooxEngine.MeshPhysicsData`1[D].<EnsureUpToDate>b__30_0 () [0x000f3] in <d79cbb467f0147628a85804f9bdaa366>:0 <---

   at void BaseX.UniLog.Error(string message, bool stackTrace)
   at void FrooxEngine.CoroutineManager.CheckExceptions(Task task)
   at void System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke() in /home/ben/build/mono-git/src/mono/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 48
   at void System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() in /home/ben/build/mono-git/src/mono/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2334
   at bool System.Threading.ThreadPoolWorkQueue.Dispatch() in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:line 785
   at bool System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:line 1261
4:41:14 PM.098  Exception running asynchronous task:
System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object) ---> System.NullReferenceException: Object reference not set to an instance of an object
  at BepuUtilities.Collections.QuickSort.Sort[TKey,TComparer] (TKey& keys, System.Int32 l, System.Int32 r, TComparer& comparer) <0x4368bfd0 + 0x0006f> in <9b84f04af68846b0bf7215c07b3361e5>:0 
  at BepuPhysics.Trees.Tree.FindPartitionForAxis (BepuUtilities.BoundingBox* boundingBoxes, BepuUtilities.BoundingBox* aMerged, System.Single* centroids, System.Int32* indexMap, System.Int32 count, System.Int32& splitIndex, System.Single& cost, BepuUtilities.BoundingBox& a, BepuUtilities.BoundingBox& b, System.Int32& leafCountA, System.Int32& leafCountB) <0x4368b9f0 + 0x00072> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.FindPartition (BepuPhysics.Trees.Tree+SweepResources& leaves, System.Int32 start, System.Int32 count, System.Int32& splitIndex, BepuUtilities.BoundingBox& a, BepuUtilities.BoundingBox& b, System.Int32& leafCountA, System.Int32& leafCountB) <0x4368b340 + 0x00194> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.SplitLeavesIntoChildren (BepuPhysics.Trees.Tree+SweepResources& leaves, System.Int32 start, System.Int32 count, System.Int32 nodeIndex) <0x4368af80 + 0x000a6> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.CreateSweepBuilderNode (System.Int32 parentIndex, System.Int32 indexInParent, BepuPhysics.Trees.Tree+SweepResources& leaves, System.Int32 start, System.Int32 count) <0x4368acc0 + 0x00075> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.SweepBuild (BepuUtilities.Memory.BufferPool pool, BepuUtilities.Memory.Buffer`1[T] leafBounds) <0x43689ca0 + 0x0057b> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Collidables.Mesh..ctor (BepuUtilities.Memory.Buffer`1[T] triangles, System.Numerics.Vector3& scale, BepuUtilities.Memory.BufferPool pool) <0x43689730 + 0x0029c> in <f1c97308329249378dafd71e68e14cb9>:0 
  at CodeX.BepuMeshHelper.ComputeMeshCollider (BaseX.MeshX meshx, System.Boolean dualSided, BepuUtilities.Memory.BufferPool bufferPool, System.Object bufferLock, System.Action meshDataExtracted, BepuPhysics.Collidables.Mesh& mesh) <0x436817c0 + 0x0013d> in <368a2654c3714372b779a48d890211a7>:0 
  at FrooxEngine.MeshColliderData.BuildData () [0x000c1] in <d79cbb467f0147628a85804f9bdaa366>:0 
  at FrooxEngine.MeshPhysicsData`1[D].<EnsureUpToDate>b__30_0 () [0x000f3] in <d79cbb467f0147628a85804f9bdaa366>:0 
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object
  at BepuUtilities.Collections.QuickSort.Sort[TKey,TComparer] (TKey& keys, System.Int32 l, System.Int32 r, TComparer& comparer) <0x4368bfd0 + 0x0006f> in <9b84f04af68846b0bf7215c07b3361e5>:0 
  at BepuPhysics.Trees.Tree.FindPartitionForAxis (BepuUtilities.BoundingBox* boundingBoxes, BepuUtilities.BoundingBox* aMerged, System.Single* centroids, System.Int32* indexMap, System.Int32 count, System.Int32& splitIndex, System.Single& cost, BepuUtilities.BoundingBox& a, BepuUtilities.BoundingBox& b, System.Int32& leafCountA, System.Int32& leafCountB) <0x4368b9f0 + 0x00072> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.FindPartition (BepuPhysics.Trees.Tree+SweepResources& leaves, System.Int32 start, System.Int32 count, System.Int32& splitIndex, BepuUtilities.BoundingBox& a, BepuUtilities.BoundingBox& b, System.Int32& leafCountA, System.Int32& leafCountB) <0x4368b340 + 0x00194> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.SplitLeavesIntoChildren (BepuPhysics.Trees.Tree+SweepResources& leaves, System.Int32 start, System.Int32 count, System.Int32 nodeIndex) <0x4368af80 + 0x000a6> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.CreateSweepBuilderNode (System.Int32 parentIndex, System.Int32 indexInParent, BepuPhysics.Trees.Tree+SweepResources& leaves, System.Int32 start, System.Int32 count) <0x4368acc0 + 0x00075> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.SweepBuild (BepuUtilities.Memory.BufferPool pool, BepuUtilities.Memory.Buffer`1[T] leafBounds) <0x43689ca0 + 0x0057b> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Collidables.Mesh..ctor (BepuUtilities.Memory.Buffer`1[T] triangles, System.Numerics.Vector3& scale, BepuUtilities.Memory.BufferPool pool) <0x43689730 + 0x0029c> in <f1c97308329249378dafd71e68e14cb9>:0 
  at CodeX.BepuMeshHelper.ComputeMeshCollider (BaseX.MeshX meshx, System.Boolean dualSided, BepuUtilities.Memory.BufferPool bufferPool, System.Object bufferLock, System.Action meshDataExtracted, BepuPhysics.Collidables.Mesh& mesh) <0x436817c0 + 0x0013d> in <368a2654c3714372b779a48d890211a7>:0 
  at FrooxEngine.MeshColliderData.BuildData () [0x000c1] in <d79cbb467f0147628a85804f9bdaa366>:0 
  at FrooxEngine.MeshPhysicsData`1[D].<EnsureUpToDate>b__30_0 () [0x000f3] in <d79cbb467f0147628a85804f9bdaa366>:0 <---

   at void BaseX.UniLog.Error(string message, bool stackTrace)
   at void FrooxEngine.CoroutineManager.CheckExceptions(Task task)
   at void System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke() in /home/ben/build/mono-git/src/mono/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 48
   at void System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() in /home/ben/build/mono-git/src/mono/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2334
   at bool System.Threading.ThreadPoolWorkQueue.Dispatch() in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:line 785
   at bool System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:line 1261
4:41:15 PM.211  MaxUsers set to 40. LastModifyingUser: 

   at void BaseX.UniLog.Log(string message, bool stackTrace)
   at void FrooxEngine.WorldConfiguration.FieldChanged(IChangeable obj)
   at void FrooxEngine.SyncElement.SyncElementChanged(IChangeable member)
   at void FrooxEngine.SyncField<int>.ValueChanged()
   at bool FrooxEngine.SyncField<int>.InternalSetValue(in int value, bool sync, bool change)
   at void FrooxEngine.SyncField<int>.set_Value(int value)
   at void FrooxEngine.Sync<int>.set_Value(int value)
   at void FrooxEngine.World.set_MaxUsers(int value)+() => { }
   at void FrooxEngine.World.RunSynchronousActions()
   at void FrooxEngine.World.NextRefreshStage()
   at bool FrooxEngine.World.Refresh()
   at void FrooxEngine.WorldManager.UpdateStep(double maxMilliseconds)
   at bool FrooxEngine.WorldManager.RunUpdateLoop(double maxMilliseconds)
   at void FrooxEngine.Engine.UpdateStep(double maxMilliseconds)
   at void FrooxEngine.Engine.RunUpdateLoop(double maxMilliseconds)
   at void NeosHeadless.Program.UpdateLoop()
   at void System.Threading.ThreadHelper.ThreadStart_Context(object state) in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:line 71
   at void System.Threading.ThreadHelper.ThreadStart() in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:line 109
4:41:15 PM.212  WorldDescription set to . LastModifyingUser: 

   at void BaseX.UniLog.Log(string message, bool stackTrace)
   at void FrooxEngine.WorldConfiguration.FieldChanged(IChangeable obj)
   at void FrooxEngine.SyncElement.SyncElementChanged(IChangeable member)
   at void FrooxEngine.SyncField<string>.ValueChanged()
   at bool FrooxEngine.SyncField<string>.InternalSetValue(in string value, bool sync, bool change)
   at void FrooxEngine.SyncField<string>.set_Value(string value)
   at void FrooxEngine.Sync<string>.set_Value(string value)
   at void FrooxEngine.World.set_Description(string value)+() => { }
   at void FrooxEngine.World.RunSynchronousActions()
   at void FrooxEngine.World.NextRefreshStage()
   at bool FrooxEngine.World.Refresh()
   at void FrooxEngine.WorldManager.UpdateStep(double maxMilliseconds)
   at bool FrooxEngine.WorldManager.RunUpdateLoop(double maxMilliseconds)
   at void FrooxEngine.Engine.UpdateStep(double maxMilliseconds)
   at void FrooxEngine.Engine.RunUpdateLoop(double maxMilliseconds)
   at void NeosHeadless.Program.UpdateLoop()
   at void System.Threading.ThreadHelper.ThreadStart_Context(object state) in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:line 71
   at void System.Threading.ThreadHelper.ThreadStart() in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:line 109
4:41:16 PM.088  World running...
6:35:58 PM.473  Exception in background job:
System.NullReferenceException: Object reference not set to an instance of an object
  at BepuUtilities.Collections.QuickList`1[T].Resize (System.Int32 newSize, BepuUtilities.Memory.IUnmanagedMemoryPool pool) <0x44193110 + 0x0004b> in <9b84f04af68846b0bf7215c07b3361e5>:0 
  at BepuPhysics.Trees.Tree.RefitAndMark (System.Int32 leafCountThreshold, BepuUtilities.Collections.QuickList`1[System.Int32]& refinementCandidates, BepuUtilities.Memory.BufferPool pool) <0x44e566a0 + 0x00112> in <f1c97308329249378dafd71e68e14cb9>:0 
  at BepuPhysics.Trees.Tree.RefitAndRefine (BepuUtilities.Memory.BufferPool pool, System.Int32 frameIndex, System.Single refineAggressivenessScale, System.Single cacheOptimizeAggressivenessScale) <0x4189cfc0 + 0x00177> in <f1c97308329249378dafd71e68e14cb9>:0 
  at FrooxEngine.BoundingBoxTree.RefitAndRefine () <0x423669a0 + 0x0003f> in <d79cbb467f0147628a85804f9bdaa366>:0 
  at FrooxEngine.DynamicBoneChainManager.UpdateAccelerationStructures () <0x42365e80 + 0x002be> in <d79cbb467f0147628a85804f9bdaa366>:0 
  at BaseX.WorkProcessor.ProcessQueue (BaseX.WorkProcessor+ThreadWorker worker, System.Collections.Concurrent.ConcurrentQueue`1[T] queue) [0x00009] in <77881a2cfe8f46e6b06b559ee82b4ed3>:0

Full log - https://neos.xnode.org/tmp/gh2914-2.log

Enverex commented 3 years ago

The headless is pretty much unusable at this point, it's dying within a few hours of being brought online each time.

Latest full log example - https://neos.xnode.org/tmp/gh2914-3.log

Frooxius commented 3 years ago

Did you notice this happening due to a particular world? Or any of them? I can add a bit of extra diagnostics and see if we can narrow it down some, but if you can tell which world it's happening in, that can help.

Enverex commented 3 years ago

I believe it was users in the world, not the worlds themselves triggering it so I couldn't see anywhere in particular. You've just pushed a few dynamic bone crash fixes though so I'll see if the issue disappears now and report back.

Frooxius commented 3 years ago

Hmm... Give it a test with the latest 2021.9.3.1281 and see if that makes a difference. It might've been the dynamic bones perhaps, but I'm not sure at this point. If it still occurs, I'll add more diagnostics.

Enverex commented 3 years ago

Unfortunately still seems to be happening. Full log - https://neos.xnode.org/tmp/gh2914-4.log

Frooxius commented 3 years ago

I found an issue in a very dense mesh, it seems like it was trying to compute a collider for it and that might've caused some corruption. I pushed a fix in 2021.9.4.468.

Please give that a try and see if the issue is still there.

What version of Mono are you using by the way though? The stack traces are a bit bizarre and don't actually match the Neos code. Are you using something like LLVM or alternate JIT?

Enverex commented 3 years ago

It looks like Mono got switched to using LLVM at some point, which should apparently be faster, but I'm guessing it's tripping something up in Neos and that's causing problems?

Also could you tell from the logs where that collider was? There shouldn't be anything in any of my worlds that is anywhere near that high poly.

Frooxius commented 3 years ago

I'm not sure how stable the LLVM JIT is, it might not be generating correct code in some cases and that might be causing issues.

Don't know where the collider was, it was some kind of terrain that looked like a cave or sea floor. I'd have to search around. It was about ~14M triangles, it was tripping up on the dual sided version of the collider.

Enverex commented 3 years ago

I can't find much information on the Mono LLVM JIT being incomplete or unstable so I'm interested to know if it's triggering things in Neos that need to be fixed, or if it's something that needs to be reported back to Mono upstream. Apparently LLVM JIT should give a substantial performance boost to Mono so it would be beneficial for it to work if at all possible.

Me and Matty spent an hour or so going around all the maps that the headless hosts trying to find that problematic mesh but there's nothing even remotely close. The closest thing is a ~500,000 poly terrain in one map, but that's a fraction of the number and was already set to front only so if you do ever manage to find what it is (or can provide me with the ID) it would be appreciated as I'd obviously like to get rid of it or at least figure out why it exists.

Frooxius commented 3 years ago

Usually compiler bugs like that are very subtle and devious and you won't too much on them online. Best thing is to check if that's actually source of problems - run for a while with LLVM JIT and then with regular JIT and see if problems disappear.

For diagnosing issues like this, can you use regular JIT please? With the LLVM I can't match up the stack trace to the actual code, which makes finding the cause significantly more difficult.

Enverex commented 3 years ago

I switched back to non-LLVM after my last comment and the headless hasn't become unresponsive again since.

I can switch to LLVM on the fly so if you'd like me to test anything with that specifically going forward, please let me know.

Frooxius commented 3 years ago

Ah good to know! If you could switch to LLVM for a bit that could help, we could confirm if it was source of the issues or if we actually fixed the issue.

Enverex commented 3 years ago

Tried it again with LLVM, joined one of the worlds and Mono core dumped (no output from Neos this time).

=================================================================
        Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
/home/steam/NeosVR/runNeos.sh: line 11: 1491947 Aborted                 (core dumped) mono --llvm Neos.exe

So nothing particularly useful there. Normally it's not so abrupt and at least allows Neos itself to crash.

Enverex commented 3 years ago

Looks like I spoke to soon. Headless became unresponsive after about 2 days. Neos was still running, but wasn't returning a new shell or responding to carriage returns. Nothing had been output to the terminal since a world had restarted though there's a suspicious listing of NaNs.

image

Same terminal output in text format:

User Joined Blueberry Smoothie Club. Username: EnverexBot, UserID: U-EnverexBot, MachineID: 3eqxf_36huqlpjlpqvqlza
User Spawn Blueberry Smoothie Club. Username: EnverexBot, UserID: U-EnverexBot, MachineID: 3eqxf_36huqlpjlpqvqlza
Default Host Role: Admin
Exception trying to load video URL https://youtube.com/watch?v=QIOn6JyOG-I from video service:
System.IO.FileNotFoundException: youtube-dl not found!
  at NYoutubeDL.Services.PreparationService.SetupPrepare (NYoutubeDL.YoutubeDL ydl) [0x000de] in <11f13f42977d43ebb3f898315a1fa2fd>:0
  at NYoutubeDL.Services.InfoService.GetDownloadInfoAsync (NYoutubeDL.YoutubeDL ydl, System.Threading.CancellationToken cancellationToken) [0x000d8] in <11f13f42977d43ebb3f898315a1fa2fd>:0
  at NYoutubeDL.Services.InfoService.GetDownloadInfoAsync (NYoutubeDL.YoutubeDL ydl, System.String url, System.Threading.CancellationToken cancellationToken) [0x00077] in <11f13f42977d43ebb3f898315a1fa2fd>:0
  at NYoutubeDL.YoutubeDL.GetDownloadInfoAsync (System.String url) [0x000dd] in <11f13f42977d43ebb3f898315a1fa2fd>:0
  at FrooxEngine.VideoTextureProvider.LoadFromVideoServiceIntern (System.Uri url, System.Threading.CancellationToken cancellationToken) [0x000f0] in <f3a38dec6c394bcfb804878c7c0b7a9f>:0
  at FrooxEngine.VideoTextureProvider.LoadFromVideoService (System.Uri url, System.Threading.CancellationToken cancellationToken) [0x0007d] in <f3a38dec6c394bcfb804878c7c0b7a9f>:0

   at void BaseX.UniLog.Error(string message, bool stackTrace)
   at async Task FrooxEngine.VideoTextureProvider.LoadFromVideoService(Uri url, CancellationToken cancellationToken)
   at void System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<<LoadFromVideoService>d__68>(ref <LoadFromVideoService>d__68 stateMachine) in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/runtime/compilerservices/AsyncMethodBuilder.cs:line 304
   at Task FrooxEngine.VideoTextureProvider.LoadFromVideoService(Uri url, CancellationToken cancellationToken)
   at void FrooxEngine.VideoTextureProvider+<>c__DisplayClass62_0+<<UpdateAsset>b__0>d.MoveNext()
   at void System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<<<UpdateAsset>b__0>d>(ref <<UpdateAsset>b__0>d stateMachine) in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/runtime/compilerservices/AsyncMethodBuilder.cs:line 304
   at void FrooxEngine.VideoTextureProvider.UpdateAsset()+() => { } [0]
   at Task FrooxEngine.CoroutineManager.StartTask(Func<Task> task, IUpdatable updatable)
   at Task FrooxEngine.Worker.StartTask(Func<Task> task)
   at void FrooxEngine.VideoTextureProvider.UpdateAsset()
   at void FrooxEngine.AssetProvider<FrooxEngine.VideoTexture>.RefreshAssetState()
   at void FrooxEngine.AssetProvider<FrooxEngine.VideoTexture>.OnChanges()
   at void FrooxEngine.ComponentBase<FrooxEngine.Component>.InternalRunApplyChanges(int updateIndex)
   at void FrooxEngine.UpdateManager.ProcessChange(IUpdatable updatable)
   at bool FrooxEngine.UpdateManager.RunQueue<IUpdatable>(Queue<IUpdatable> queue, Action<IUpdatable> action)
   at bool FrooxEngine.UpdateManager.RunChangeApplications()
   at bool FrooxEngine.World.RefreshStep()
   at bool FrooxEngine.World.Refresh()
   at void FrooxEngine.WorldManager.UpdateStep(double maxMilliseconds)
   at bool FrooxEngine.WorldManager.RunUpdateLoop(double maxMilliseconds)
   at void FrooxEngine.Engine.UpdateStep(double maxMilliseconds)
   at void FrooxEngine.Engine.RunUpdateLoop(double maxMilliseconds)
   at void NeosHeadless.Program.UpdateLoop()
   at void System.Threading.ThreadHelper.ThreadStart_Context(object state) in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:line 71
   at void System.Threading.ThreadHelper.ThreadStart() in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:line 109
Invalid bounds: Center: [NaN; NaN; 0] Size: [NaN; NaN; 0] -- Min: [NaN; NaN; 0] Max: [NaN; NaN; 0], owner: Element: ID315FF, Type: FrooxEngine.LocalMeshProvider, World: Blueberry Smoothie Club, IsRemoved: False, IsDestroyed: False, IsDisposed: False, Enabled: True
Element: ID2F1FF, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Chunk, T: [0; 0; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID2D7FF, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Chunk, T: [0; 0; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID1E82100, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Canvas, T: [0; 0; 0], R: [-1.135147E-14; 1.080599E-09; 1.023103E-14; 1], S: [0.001; 0.001; 0.001], ActiveSelf: True, IsDestroyed: False
Element: ID1E71E00, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: RawUI, T: [0; 1000; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID1B30500, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Playlist 2.1, T: [-11.07566; 3.570425; 42.78163], R: [3.731029E-08; 0.05904979; -4.845483E-08; 0.9982551], S: [0.3242003; 0.334227; 0.3342269], ActiveSelf: True, IsDestroyed: False
Element: ID544F00, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: World Root, T: [0; 0; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID2100, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Root, T: [0; 0; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID0, Type: FrooxEngine.World, World: Blueberry Smoothie Club, IsRemoved: False
, URL:
Members on: FrooxEngine.LocalMeshProvider - ID315FF
persistent - FrooxEngine.Sync`1[System.Boolean] - ID315FF: True
UpdateOrder - FrooxEngine.Sync`1[System.Int32] - ID315FF: 0
Enabled - FrooxEngine.Sync`1[System.Boolean] - ID315FF: True
HighPriorityIntegration - FrooxEngine.Sync`1[System.Boolean] - ID315FF: False

Invalid bounds: Center: [960; NaN; 0] Size: [1920; NaN; 0] -- Min: [0; NaN; 0] Max: [1920; NaN; 0], owner: Element: ID2FCFF, Type: FrooxEngine.LocalMeshProvider, World: Blueberry Smoothie Club, IsRemoved: False, IsDestroyed: False, IsDisposed: False, Enabled: True
Element: ID2F1FF, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Chunk, T: [0; 0; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID2D7FF, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Chunk, T: [0; 0; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID1E82100, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Canvas, T: [0; 0; 0], R: [-1.135147E-14; 1.080599E-09; 1.023103E-14; 1], S: [0.001; 0.001; 0.001], ActiveSelf: True, IsDestroyed: False
Element: ID1E71E00, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: RawUI, T: [0; 1000; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID1B30500, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Playlist 2.1, T: [-11.07566; 3.570425; 42.78163], R: [3.731029E-08; 0.05904979; -4.845483E-08; 0.9982551], S: [0.3242003; 0.334227; 0.3342269], ActiveSelf: True, IsDestroyed: False
Element: ID544F00, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: World Root, T: [0; 0; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID2100, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Root, T: [0; 0; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID0, Type: FrooxEngine.World, World: Blueberry Smoothie Club, IsRemoved: False
, URL:
Members on: FrooxEngine.LocalMeshProvider - ID2FCFF
persistent - FrooxEngine.Sync`1[System.Boolean] - ID2FCFF: True
UpdateOrder - FrooxEngine.Sync`1[System.Int32] - ID2FCFF: 0
Enabled - FrooxEngine.Sync`1[System.Boolean] - ID2FCFF: True
HighPriorityIntegration - FrooxEngine.Sync`1[System.Boolean] - ID2FCFF: False

Invalid bounds: Center: [960; NaN; 0] Size: [1920; NaN; 0] -- Min: [0; NaN; 0] Max: [1920; NaN; 0], owner: Element: ID2E2FF, Type: FrooxEngine.LocalMeshProvider, World: Blueberry Smoothie Club, IsRemoved: False, IsDestroyed: False, IsDisposed: False, Enabled: True
Element: ID2D7FF, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Chunk, T: [0; 0; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID1E82100, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Canvas, T: [0; 0; 0], R: [-1.135147E-14; 1.080599E-09; 1.023103E-14; 1], S: [0.001; 0.001; 0.001], ActiveSelf: True, IsDestroyed: False
Element: ID1E71E00, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: RawUI, T: [0; 1000; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID1B30500, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Playlist 2.1, T: [-11.07566; 3.570425; 42.78163], R: [3.731029E-08; 0.05904979; -4.845483E-08; 0.9982551], S: [0.3242003; 0.334227; 0.3342269], ActiveSelf: True, IsDestroyed: False
Element: ID544F00, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: World Root, T: [0; 0; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID2100, Type: FrooxEngine.Slot, World: Blueberry Smoothie Club, IsRemoved: False, Slot name: Root, T: [0; 0; 0], R: [0; 0; 0; 1], S: [1; 1; 1], ActiveSelf: True, IsDestroyed: False
Element: ID0, Type: FrooxEngine.World, World: Blueberry Smoothie Club, IsRemoved: False
, URL:
Members on: FrooxEngine.LocalMeshProvider - ID2E2FF
persistent - FrooxEngine.Sync`1[System.Boolean] - ID2E2FF: True
UpdateOrder - FrooxEngine.Sync`1[System.Int32] - ID2E2FF: 0
Enabled - FrooxEngine.Sync`1[System.Boolean] - ID2E2FF: True
HighPriorityIntegration - FrooxEngine.Sync`1[System.Boolean] - ID2E2FF: False

MaxUsers set to 40. LastModifyingUser:

   at void BaseX.UniLog.Log(string message, bool stackTrace)
   at void FrooxEngine.WorldConfiguration.FieldChanged(IChangeable obj)
   at void FrooxEngine.SyncElement.SyncElementChanged(IChangeable member)
   at void FrooxEngine.SyncField<int>.ValueChanged()
   at bool FrooxEngine.SyncField<int>.InternalSetValue(in int value, bool sync, bool change)
   at void FrooxEngine.SyncField<int>.set_Value(int value)
   at void FrooxEngine.Sync<int>.set_Value(int value)
   at void FrooxEngine.World.set_MaxUsers(int value)+() => { }
   at void FrooxEngine.World.RunSynchronousActions()
   at void FrooxEngine.World.NextRefreshStage()
   at bool FrooxEngine.World.Refresh()
   at void FrooxEngine.WorldManager.UpdateStep(double maxMilliseconds)
   at bool FrooxEngine.WorldManager.RunUpdateLoop(double maxMilliseconds)
   at void FrooxEngine.Engine.UpdateStep(double maxMilliseconds)
   at void FrooxEngine.Engine.RunUpdateLoop(double maxMilliseconds)
   at void NeosHeadless.Program.UpdateLoop()
   at void System.Threading.ThreadHelper.ThreadStart_Context(object state) in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:line 71
   at void System.Threading.ThreadHelper.ThreadStart() in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:line 109
WorldDescription set to . LastModifyingUser:

   at void BaseX.UniLog.Log(string message, bool stackTrace)
   at void FrooxEngine.WorldConfiguration.FieldChanged(IChangeable obj)
   at void FrooxEngine.SyncElement.SyncElementChanged(IChangeable member)
   at void FrooxEngine.SyncField<string>.ValueChanged()
   at bool FrooxEngine.SyncField<string>.InternalSetValue(in string value, bool sync, bool change)
   at void FrooxEngine.SyncField<string>.set_Value(string value)
   at void FrooxEngine.Sync<string>.set_Value(string value)
   at void FrooxEngine.World.set_Description(string value)+() => { }
   at void FrooxEngine.World.RunSynchronousActions()
   at void FrooxEngine.World.NextRefreshStage()
   at bool FrooxEngine.World.Refresh()
   at void FrooxEngine.WorldManager.UpdateStep(double maxMilliseconds)
   at bool FrooxEngine.WorldManager.RunUpdateLoop(double maxMilliseconds)
   at void FrooxEngine.Engine.UpdateStep(double maxMilliseconds)
   at void FrooxEngine.Engine.RunUpdateLoop(double maxMilliseconds)
   at void NeosHeadless.Program.UpdateLoop()
   at void System.Threading.ThreadHelper.ThreadStart_Context(object state) in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:line 71
   at void System.Threading.ThreadHelper.ThreadStart() in /home/ben/build/mono-git/src/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:line 109
World running...
Blueberry Smoothie Club>

And the full log - https://neos.xnode.org/tmp/gh2914-5.log

Frooxius commented 3 years ago

Unfortunately I can't find anything in the log that would indicate the cause. The NaN's there are harmless, they're just for meshes.

Does this happen to a specific world perhaps? Or something else that could narrow it down? It's odd, because it doesn't seem to throw exception of any kind or anything that would get logged.