search

space-wizards / RobustToolbox

Robust multiplayer game engine, used by Space Station 14
https://spacestation14.io
Other
553 stars 414 forks source link

EntityLookup exceptions #4211

Open metalgearsloth opened 1 year ago

metalgearsloth commented 1 year ago

Looks like there were a few different exceptions that led to a server crash (times are AEST).

First exception image

Last exception image

metalgearsloth commented 1 year ago

Okay looks like it's mostly from PA shitcode so probably best to start there.

ElectroJr commented 1 year ago

There are also three related exceptions that have happened recently, two of which are NRE exceptions in different functions.

Found null broadphase for ShadyCigs Deluxe (1793330, VendingMachineCigs)
 Sawmill=system.broad_phase
Caught exception in entsys
System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Robust.Shared.Physics.BroadPhase.DynamicTreeBroadPhase.AabbQueryStateCallback[TState](ValueTuple`6& tuple, Proxy proxy) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/BroadPhase/DynamicTreeBroadPhase.cs:line 136
   at Robust.Shared.Physics.B2DynamicTree`1.Query[TState](TState& state, QueryCallback`1 callback, Box2& aabb) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 894
   at Robust.Shared.Physics.BroadPhase.DynamicTreeBroadPhase.QueryAabb[TState](TState& state, QueryCallbackDelegate`1 callback, Box2 aabb, Boolean approx) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/BroadPhase/DynamicTreeBroadPhase.cs:line 64
   at Robust.Shared.Physics.Systems.SharedBroadphaseSystem.QueryBroadphase(IBroadPhase broadPhase, ValueTuple`2 state, Box2 aabb) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/Systems/SharedBroadphaseSystem.cs:line 413
   at Robust.Shared.Physics.Systems.SharedBroadphaseSystem.FindPairs(FixtureProxy proxy, Box2 worldAABB, EntityUid broadphase, List`1 pairBuffer) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/Systems/SharedBroadphaseSystem.cs:line 397
   at Robust.Shared.Map.MapManager.<>c__61`1.<FindGridsIntersecting>b__61_0(ValueTuple`7& tuple, Proxy proxy) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Map/MapManager.Queries.cs:line 86
   at Robust.Shared.Physics.B2DynamicTree`1.Query[TState](TState& state, QueryCallback`1 callback, Box2& aabb) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 894
   at Robust.Shared.Map.MapManager.FindGridsIntersecting[TState](MapId mapId, Box2 worldAABB, TState& state, GridCallback`1 callback, Boolean approx, Boolean includeMap) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Map/MapManager.Queries.cs:line 71
   at Robust.Shared.Physics.Systems.SharedBroadphaseSystem.<>c__DisplayClass21_0.<FindNewContacts>b__0(Int32 i) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/Systems/SharedBroadphaseSystem.cs:line 204
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.For(Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body)
   at Robust.Shared.Physics.Systems.SharedBroadphaseSystem.FindNewContacts(PhysicsMapComponent component, MapId mapId)
   at Robust.Shared.Physics.Systems.SharedPhysicsSystem.SimulateWorld(Single deltaTime, Boolean prediction) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/Systems/SharedPhysicsSystem.cs:line 298
   at Robust.Shared.GameObjects.EntitySystemManager.TickUpdate(Single frameTime, Boolean noPredictions) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntitySystemManager.cs:line 318
 Catcher=entsys Sawmill=runtime

And finally:

Job failed on exception:
System.NullReferenceException: Object reference not set to an instance of an object.
   at Robust.Shared.Physics.BroadPhase.DynamicTreeBroadPhase.ExtractAabbFunc(FixtureProxy& proxy) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/BroadPhase/DynamicTreeBroadPhase.cs:line 23
   at Robust.Shared.Physics.BroadPhase.DynamicTreeBroadPhase.AabbQueryStateCallback[TState](ValueTuple`6& tuple, Proxy proxy) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/BroadPhase/DynamicTreeBroadPhase.cs:line 136
   at Robust.Shared.Physics.B2DynamicTree`1.Query[TState](TState& state, QueryCallback`1 callback, Box2& aabb) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 894
   at Robust.Shared.Physics.BroadPhase.DynamicTreeBroadPhase.QueryAabb[TState](TState& state, QueryCallbackDelegate`1 callback, Box2 aabb, Boolean approx) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/BroadPhase/DynamicTreeBroadPhase.cs:line 64
   at Robust.Shared.GameObjects.EntityLookupSystem.AddComponentsIntersecting[T](EntityUid lookupUid, HashSet`1 intersecting, Box2 worldAABB, LookupFlags flags, EntityQuery`1 lookupQuery, EntityQuery`1 xformQuery, EntityQuery`1 query) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.ComponentQueries.cs:line 28
   at Robust.Shared.GameObjects.EntityLookupSystem.GetComponentsIntersecting(Type type, MapId mapId, Box2 worldAABB, LookupFlags flags) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.ComponentQueries.cs:line 240
   at Robust.Shared.GameObjects.EntityLookupSystem.GetComponentsInRange(Type type, MapId mapId, Vector2 worldPos, Single range) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.ComponentQueries.cs:line 332
   at Content.Server.NPC.Systems.NPCUtilitySystem.Add(NPCBlackboard blackboard, HashSet`1 entities, UtilityQuery query) in /home/runner/work/space-station-14/space-station-14/Content.Server/NPC/Systems/NPCUtilitySystem.cs:line 302
   at Content.Server.NPC.Systems.NPCUtilitySystem.GetEntities(NPCBlackboard blackboard, String proto, Boolean bestOnly) in /home/runner/work/space-station-14/space-station-14/Content.Server/NPC/Systems/NPCUtilitySystem.cs:line 49
   at Content.Server.NPC.HTN.PrimitiveTasks.Operators.UtilityOperator.Plan(NPCBlackboard blackboard, CancellationToken cancelToken) in /home/runner/work/space-station-14/space-station-14/Content.Server/NPC/HTN/PrimitiveTasks/Operators/UtilityOperator.cs:line 47
   at Content.Server.NPC.HTN.HTNPlanJob.PrimitiveConditionMet(HTNPrimitiveTask primitive, NPCBlackboard blackboard, List`1 appliedStates) in /home/runner/work/space-station-14/space-station-14/Content.Server/NPC/HTN/HTNPlanJob.cs:line 155
   at Robust.Shared.CPUJob.JobQueues.Job`1.WaitAsyncTask[TTask](Task`1 task) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/CPUJob/JobQueues/Job.cs:line 107
   at Content.Server.NPC.HTN.HTNPlanJob.Process() in /home/runner/work/space-station-14/space-station-14/Content.Server/NPC/HTN/HTNPlanJob.cs:line 123
   at Robust.Shared.CPUJob.JobQueues.Job`1.ProcessWrap() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/CPUJob/JobQueues/Job.cs:line 201
 Sawmill=job

I'm also going to repost the other two exceptions so that they're searchable / not images

Caught exception in entsys
System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at Robust.Shared.Physics.B2DynamicTree`1.Balance(Proxy index)
   at Robust.Shared.Physics.B2DynamicTree`1.RemoveLeaf(Proxy leaf) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 392
   at Robust.Shared.GameObjects.EntityLookupSystem.RemoveBroadTree(BroadphaseComponent lookup, FixturesComponent manager, Boolean staticBody, PhysicsMapComponent map) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 317
   at Robust.Shared.GameObjects.EntityLookupSystem.UpdatePhysicsBroadphase(EntityUid uid, TransformComponent xform, PhysicsComponent body) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 283
   at Robust.Shared.GameObjects.EntityEventBus.ProcessSingleEventCore(EventSource source, Unit& unitRef, EventData subs, Boolean byRef) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Broadcast.cs:line 318
   at Robust.Shared.GameObjects.EntityEventBus.RaiseEvent[T](EventSource source, T& toRaise) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Broadcast.cs:line 271
   at Content.Shared.Doors.Systems.SharedDoorSystem.NextState(DoorComponent door, TimeSpan time) in /home/runner/work/space-station-14/space-station-14/Content.Shared/Doors/Systems/SharedDoorSystem.cs:line 611
   at Content.Shared.Doors.Systems.SharedDoorSystem.Update(Single frameTime) in /home/runner/work/space-station-14/space-station-14/Content.Shared/Doors/Systems/SharedDoorSystem.cs:line 578
   at Robust.Shared.GameObjects.EntitySystemManager.TickUpdate(Single frameTime, Boolean noPredictions) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntitySystemManager.cs:line 318
 Catcher=entsys Sawmill=runtime
Caught exception in entsys
System.Exception: Infinite loop in B2DynamicTree.Balance(). Trace:    at System.Environment.get_StackTrace()
   at Robust.Shared.Physics.B2DynamicTree`1.Balance(Proxy index) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 586
   at Robust.Shared.Physics.B2DynamicTree`1.RemoveLeaf(Proxy leaf) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 426
   at Robust.Shared.Physics.B2DynamicTree`1.MoveProxy(Proxy proxy, Box2& aabb, Vector2 displacement) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 343
   at Robust.Shared.GameObjects.EntityLookupSystem.AddOrUpdatePhysicsTree(EntityUid uid, EntityUid broadUid, BroadphaseComponent broadphase, TransformComponent broadphaseXform, PhysicsMapComponent physicsMap, TransformComponent xform, PhysicsComponent body, FixturesComponent manager, EntityQuery`1 xformQuery) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 379
   at Robust.Shared.GameObjects.EntityLookupSystem.AddOrUpdateEntityTree(EntityUid broadUid, BroadphaseComponent broadphase, TransformComponent broadphaseXform, PhysicsMapComponent physicsMap, EntityUid uid, TransformComponent xform, Boolean recursive) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 663
   at Robust.Shared.GameObjects.EntityLookupSystem.OnMove(MoveEvent& args) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 459
   at Robust.Shared.GameObjects.EntityEventBus.ProcessSingleEventCore(EventSource source, Unit& unitRef, EventData subs, Boolean byRef) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Broadcast.cs:line 324
   at Robust.Shared.GameObjects.EntityEventBus.RaiseLocalEvent[TEvent](EntityUid uid, TEvent& args, Boolean broadcast) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 200
   at Robust.Shared.GameObjects.TransformComponent.set_LocalRotation(Angle value) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Components/Transform/TransformComponent.cs:line 142
   at Content.Server.Solar.EntitySystems.PowerSolarSystem.Update(Single frameTime) in /home/runner/work/space-station-14/space-station-14/Content.Server/Solar/EntitySystems/PowerSolarSystem.cs:line 111
   at Robust.Shared.GameObjects.EntitySystemManager.TickUpdate(Single frameTime, Boolean noPredictions) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntitySystemManager.cs:line 303
   at Robust.Shared.GameObjects.EntityManager.TickUpdate(Single frameTime, Boolean noPredictions, Histogram histogram) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 267
   at Robust.Server.GameObjects.ServerEntityManager.TickUpdate(Single frameTime, Boolean noPredictions, Histogram histogram) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/GameObjects/ServerEntityManager.cs:line 153
   at Robust.Server.BaseServer.Update(FrameEventArgs frameEventArgs) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/BaseServer.cs:line 686
   at Robust.Shared.Timing.GameLoop.Run() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Timing/GameLoop.cs:line 228
   at Robust.Server.BaseServer.MainLoop() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/BaseServer.cs:line 557
   at Robust.Server.Program.ParsedMain(CommandLineArgs args, Boolean contentStart, ServerOptions options) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/Program.cs:line 78
   at Robust.Server.Program.Start(String[] args, ServerOptions options, Boolean contentStart) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/Program.cs:line 46
   at Robust.Server.Program.Main(String[] args) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/Program.cs:line 25
   at Robust.Shared.Physics.B2DynamicTree`1.Balance(Proxy index)
   at Robust.Shared.Physics.B2DynamicTree`1.RemoveLeaf(Proxy leaf) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 392
   at Robust.Shared.Physics.B2DynamicTree`1.MoveProxy(Proxy proxy, Box2& aabb, Vector2 displacement) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/Physics/B2DynamicTree.cs:line 303
   at Robust.Shared.GameObjects.EntityLookupSystem.AddOrUpdatePhysicsTree(EntityUid uid, EntityUid broadUid, BroadphaseComponent broadphase, TransformComponent broadphaseXform, PhysicsMapComponent physicsMap, TransformComponent xform, PhysicsComponent body, FixturesComponent manager, EntityQuery`1 xformQuery) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 367
   at Robust.Shared.GameObjects.EntityLookupSystem.AddOrUpdateEntityTree(EntityUid broadUid, BroadphaseComponent broadphase, TransformComponent broadphaseXform, PhysicsMapComponent physicsMap, EntityUid uid, TransformComponent xform, Boolean recursive) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 663
   at Robust.Shared.GameObjects.EntityLookupSystem.OnMove(MoveEvent& args) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Systems/EntityLookupSystem.cs:line 459
   at Robust.Shared.GameObjects.EntityEventBus.ProcessSingleEventCore(EventSource source, Unit& unitRef, EventData subs, Boolean byRef) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Broadcast.cs:line 318
   at Robust.Shared.GameObjects.EntityEventBus.RaiseLocalEvent[TEvent](EntityUid uid, TEvent& args, Boolean broadcast) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 197
   at Robust.Shared.GameObjects.TransformComponent.set_LocalRotation(Angle value) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/Components/Transform/TransformComponent.cs:line 145
   at Content.Server.Solar.EntitySystems.PowerSolarSystem.Update(Single frameTime) in /home/runner/work/space-station-14/space-station-14/Content.Server/Solar/EntitySystems/PowerSolarSystem.cs:line 118
   at Robust.Shared.GameObjects.EntitySystemManager.TickUpdate(Single frameTime, Boolean noPredictions) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntitySystemManager.cs:line 318
 Catcher=entsys Sawmill=runtime
ElectroJr commented 1 year ago

Theres a chance that the bug fixed in #4218 was the cause of these exceptions. So if they don't reoccur after an engine update we can probably close this.