search

space-wizards / RobustToolbox

Robust multiplayer game engine, used by Space Station 14
https://spacestation14.io
Other
501 stars 380 forks source link

EntityLookup exceptions #4211

Open metalgearsloth opened 11 months ago

metalgearsloth commented 11 months ago

Looks like there were a few different exceptions that led to a server crash (times are AEST).

First exception image

Last exception image

metalgearsloth commented 11 months ago

Okay looks like it's mostly from PA shitcode so probably best to start there.

ElectroJr commented 11 months ago

There are also three related exceptions that have happened recently, two of which are NRE exceptions in different functions.

Found null broadphase for ShadyCigs Deluxe (1793330, VendingMachineCigs)
 Sawmill=system.broad_phase
Caught exception in entsys
System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Robust.Shared.Physics.BroadPhase.DynamicTreeBroadPhase.AabbQueryStateCallback[TState](ValueTuple`6& tuple, Proxy proxy) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/BroadPhase/DynamicTreeBroadPhase.cs:line 136
   at Robust.Shared.Physics.B2DynamicTree`1.Query[TState](TState& state, QueryCallback`1 callback, Box2& aabb) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 894
   at Robust.Shared.Physics.BroadPhase.DynamicTreeBroadPhase.QueryAabb[TState](TState& state, QueryCallbackDelegate`1 callback, Box2 aabb, Boolean approx) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/BroadPhase/DynamicTreeBroadPhase.cs:line 64
   at Robust.Shared.Physics.Systems.SharedBroadphaseSystem.QueryBroadphase(IBroadPhase broadPhase, ValueTuple`2 state, Box2 aabb) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/Systems/SharedBroadphaseSystem.cs:line 413
   at Robust.Shared.Physics.Systems.SharedBroadphaseSystem.FindPairs(FixtureProxy proxy, Box2 worldAABB, EntityUid broadphase, List`1 pairBuffer) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/Systems/SharedBroadphaseSystem.cs:line 397
   at Robust.Shared.Map.MapManager.<>c__61`1.<FindGridsIntersecting>b__61_0(ValueTuple`7& tuple, Proxy proxy) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Map/MapManager.Queries.cs:line 86
   at Robust.Shared.Physics.B2DynamicTree`1.Query[TState](TState& state, QueryCallback`1 callback, Box2& aabb) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 894
   at Robust.Shared.Map.MapManager.FindGridsIntersecting[TState](MapId mapId, Box2 worldAABB, TState& state, GridCallback`1 callback, Boolean approx, Boolean includeMap) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Map/MapManager.Queries.cs:line 71
   at Robust.Shared.Physics.Systems.SharedBroadphaseSystem.<>c__DisplayClass21_0.<FindNewContacts>b__0(Int32 i) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/Systems/SharedBroadphaseSystem.cs:line 204
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.For(Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body)
   at Robust.Shared.Physics.Systems.SharedBroadphaseSystem.FindNewContacts(PhysicsMapComponent component, MapId mapId)
   at Robust.Shared.Physics.Systems.SharedPhysicsSystem.SimulateWorld(Single deltaTime, Boolean prediction) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/Systems/SharedPhysicsSystem.cs:line 298
   at Robust.Shared.GameObjects.EntitySystemManager.TickUpdate(Single frameTime, Boolean noPredictions) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntitySystemManager.cs:line 318
 Catcher=entsys Sawmill=runtime

And finally:

Job failed on exception:
System.NullReferenceException: Object reference not set to an instance of an object.
   at Robust.Shared.Physics.BroadPhase.DynamicTreeBroadPhase.ExtractAabbFunc(FixtureProxy& proxy) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/BroadPhase/DynamicTreeBroadPhase.cs:line 23
   at Robust.Shared.Physics.BroadPhase.DynamicTreeBroadPhase.AabbQueryStateCallback[TState](ValueTuple`6& tuple, Proxy proxy) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/BroadPhase/DynamicTreeBroadPhase.cs:line 136
   at Robust.Shared.Physics.B2DynamicTree`1.Query[TState](TState& state, QueryCallback`1 callback, Box2& aabb) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 894
   at Robust.Shared.Physics.BroadPhase.DynamicTreeBroadPhase.QueryAabb[TState](TState& state, QueryCallbackDelegate`1 callback, Box2 aabb, Boolean approx) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/BroadPhase/DynamicTreeBroadPhase.cs:line 64
   at Robust.Shared.GameObjects.EntityLookupSystem.AddComponentsIntersecting[T](EntityUid lookupUid, HashSet`1 intersecting, Box2 worldAABB, LookupFlags flags, EntityQuery`1 lookupQuery, EntityQuery`1 xformQuery, EntityQuery`1 query) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.ComponentQueries.cs:line 28
   at Robust.Shared.GameObjects.EntityLookupSystem.GetComponentsIntersecting(Type type, MapId mapId, Box2 worldAABB, LookupFlags flags) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.ComponentQueries.cs:line 240
   at Robust.Shared.GameObjects.EntityLookupSystem.GetComponentsInRange(Type type, MapId mapId, Vector2 worldPos, Single range) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.ComponentQueries.cs:line 332
   at Content.Server.NPC.Systems.NPCUtilitySystem.Add(NPCBlackboard blackboard, HashSet`1 entities, UtilityQuery query) in /home/runner/work/space-station-14/space-station-14/Content.Server/NPC/Systems/NPCUtilitySystem.cs:line 302
   at Content.Server.NPC.Systems.NPCUtilitySystem.GetEntities(NPCBlackboard blackboard, String proto, Boolean bestOnly) in /home/runner/work/space-station-14/space-station-14/Content.Server/NPC/Systems/NPCUtilitySystem.cs:line 49
   at Content.Server.NPC.HTN.PrimitiveTasks.Operators.UtilityOperator.Plan(NPCBlackboard blackboard, CancellationToken cancelToken) in /home/runner/work/space-station-14/space-station-14/Content.Server/NPC/HTN/PrimitiveTasks/Operators/UtilityOperator.cs:line 47
   at Content.Server.NPC.HTN.HTNPlanJob.PrimitiveConditionMet(HTNPrimitiveTask primitive, NPCBlackboard blackboard, List`1 appliedStates) in /home/runner/work/space-station-14/space-station-14/Content.Server/NPC/HTN/HTNPlanJob.cs:line 155
   at Robust.Shared.CPUJob.JobQueues.Job`1.WaitAsyncTask[TTask](Task`1 task) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/CPUJob/JobQueues/Job.cs:line 107
   at Content.Server.NPC.HTN.HTNPlanJob.Process() in /home/runner/work/space-station-14/space-station-14/Content.Server/NPC/HTN/HTNPlanJob.cs:line 123
   at Robust.Shared.CPUJob.JobQueues.Job`1.ProcessWrap() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/CPUJob/JobQueues/Job.cs:line 201
 Sawmill=job

I'm also going to repost the other two exceptions so that they're searchable / not images

Caught exception in entsys
System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at Robust.Shared.Physics.B2DynamicTree`1.Balance(Proxy index)
   at Robust.Shared.Physics.B2DynamicTree`1.RemoveLeaf(Proxy leaf) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 392
   at Robust.Shared.GameObjects.EntityLookupSystem.RemoveBroadTree(BroadphaseComponent lookup, FixturesComponent manager, Boolean staticBody, PhysicsMapComponent map) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 317
   at Robust.Shared.GameObjects.EntityLookupSystem.UpdatePhysicsBroadphase(EntityUid uid, TransformComponent xform, PhysicsComponent body) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 283
   at Robust.Shared.GameObjects.EntityEventBus.ProcessSingleEventCore(EventSource source, Unit& unitRef, EventData subs, Boolean byRef) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Broadcast.cs:line 318
   at Robust.Shared.GameObjects.EntityEventBus.RaiseEvent[T](EventSource source, T& toRaise) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Broadcast.cs:line 271
   at Content.Shared.Doors.Systems.SharedDoorSystem.NextState(DoorComponent door, TimeSpan time) in /home/runner/work/space-station-14/space-station-14/Content.Shared/Doors/Systems/SharedDoorSystem.cs:line 611
   at Content.Shared.Doors.Systems.SharedDoorSystem.Update(Single frameTime) in /home/runner/work/space-station-14/space-station-14/Content.Shared/Doors/Systems/SharedDoorSystem.cs:line 578
   at Robust.Shared.GameObjects.EntitySystemManager.TickUpdate(Single frameTime, Boolean noPredictions) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntitySystemManager.cs:line 318
 Catcher=entsys Sawmill=runtime
Caught exception in entsys
System.Exception: Infinite loop in B2DynamicTree.Balance(). Trace:    at System.Environment.get_StackTrace()
   at Robust.Shared.Physics.B2DynamicTree`1.Balance(Proxy index) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 586
   at Robust.Shared.Physics.B2DynamicTree`1.RemoveLeaf(Proxy leaf) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 426
   at Robust.Shared.Physics.B2DynamicTree`1.MoveProxy(Proxy proxy, Box2& aabb, Vector2 displacement) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 343
   at Robust.Shared.GameObjects.EntityLookupSystem.AddOrUpdatePhysicsTree(EntityUid uid, EntityUid broadUid, BroadphaseComponent broadphase, TransformComponent broadphaseXform, PhysicsMapComponent physicsMap, TransformComponent xform, PhysicsComponent body, FixturesComponent manager, EntityQuery`1 xformQuery) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 379
   at Robust.Shared.GameObjects.EntityLookupSystem.AddOrUpdateEntityTree(EntityUid broadUid, BroadphaseComponent broadphase, TransformComponent broadphaseXform, PhysicsMapComponent physicsMap, EntityUid uid, TransformComponent xform, Boolean recursive) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 663
   at Robust.Shared.GameObjects.EntityLookupSystem.OnMove(MoveEvent& args) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 459
   at Robust.Shared.GameObjects.EntityEventBus.ProcessSingleEventCore(EventSource source, Unit& unitRef, EventData subs, Boolean byRef) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Broadcast.cs:line 324
   at Robust.Shared.GameObjects.EntityEventBus.RaiseLocalEvent[TEvent](EntityUid uid, TEvent& args, Boolean broadcast) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 200
   at Robust.Shared.GameObjects.TransformComponent.set_LocalRotation(Angle value) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Components/Transform/TransformComponent.cs:line 142
   at Content.Server.Solar.EntitySystems.PowerSolarSystem.Update(Single frameTime) in /home/runner/work/space-station-14/space-station-14/Content.Server/Solar/EntitySystems/PowerSolarSystem.cs:line 111
   at Robust.Shared.GameObjects.EntitySystemManager.TickUpdate(Single frameTime, Boolean noPredictions) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntitySystemManager.cs:line 303
   at Robust.Shared.GameObjects.EntityManager.TickUpdate(Single frameTime, Boolean noPredictions, Histogram histogram) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 267
   at Robust.Server.GameObjects.ServerEntityManager.TickUpdate(Single frameTime, Boolean noPredictions, Histogram histogram) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/GameObjects/ServerEntityManager.cs:line 153
   at Robust.Server.BaseServer.Update(FrameEventArgs frameEventArgs) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/BaseServer.cs:line 686
   at Robust.Shared.Timing.GameLoop.Run() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Timing/GameLoop.cs:line 228
   at Robust.Server.BaseServer.MainLoop() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/BaseServer.cs:line 557
   at Robust.Server.Program.ParsedMain(CommandLineArgs args, Boolean contentStart, ServerOptions options) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/Program.cs:line 78
   at Robust.Server.Program.Start(String[] args, ServerOptions options, Boolean contentStart) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/Program.cs:line 46
   at Robust.Server.Program.Main(String[] args) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/Program.cs:line 25
   at Robust.Shared.Physics.B2DynamicTree`1.Balance(Proxy index)
   at Robust.Shared.Physics.B2DynamicTree`1.RemoveLeaf(Proxy leaf) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 392
   at Robust.Shared.Physics.B2DynamicTree`1.MoveProxy(Proxy proxy, Box2& aabb, Vector2 displacement) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 303
   at Robust.Shared.GameObjects.EntityLookupSystem.AddOrUpdatePhysicsTree(EntityUid uid, EntityUid broadUid, BroadphaseComponent broadphase, TransformComponent broadphaseXform, PhysicsMapComponent physicsMap, TransformComponent xform, PhysicsComponent body, FixturesComponent manager, EntityQuery`1 xformQuery) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 367
   at Robust.Shared.GameObjects.EntityLookupSystem.AddOrUpdateEntityTree(EntityUid broadUid, BroadphaseComponent broadphase, TransformComponent broadphaseXform, PhysicsMapComponent physicsMap, EntityUid uid, TransformComponent xform, Boolean recursive) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 663
   at Robust.Shared.GameObjects.EntityLookupSystem.OnMove(MoveEvent& args) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 459
   at Robust.Shared.GameObjects.EntityEventBus.ProcessSingleEventCore(EventSource source, Unit& unitRef, EventData subs, Boolean byRef) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Broadcast.cs:line 318
   at Robust.Shared.GameObjects.EntityEventBus.RaiseLocalEvent[TEvent](EntityUid uid, TEvent& args, Boolean broadcast) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 197
   at Robust.Shared.GameObjects.TransformComponent.set_LocalRotation(Angle value) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Components/Transform/TransformComponent.cs:line 145
   at Content.Server.Solar.EntitySystems.PowerSolarSystem.Update(Single frameTime) in /home/runner/work/space-station-14/space-station-14/Content.Server/Solar/EntitySystems/PowerSolarSystem.cs:line 118
   at Robust.Shared.GameObjects.EntitySystemManager.TickUpdate(Single frameTime, Boolean noPredictions) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntitySystemManager.cs:line 318
 Catcher=entsys Sawmill=runtime
ElectroJr commented 11 months ago

Theres a chance that the bug fixed in #4218 was the cause of these exceptions. So if they don't reoccur after an engine update we can probably close this.