moonheart08
commented
4 months ago .NET is the one supposed to be handling this. If .NET is core dumping, it's not an SS14 issue. @PJB3005
Open dffdff2423 opened 4 months ago
moonheart08
commented
4 months ago .NET is the one supposed to be handling this. If .NET is core dumping, it's not an SS14 issue. @PJB3005
dffdff2423
commented
4 months ago I have a corefile saved if that is useful. It is from an up to date fork, but the issue still exists here. It also apparently happens with the YAML linter. I have never had it happen with the client.
PJB3005
commented
4 months ago Press ctrl-c to kill it
That's a SIGINT, not a SIGTERM.
I am not able to reproduce this behavior. Both SIGTERM and SIGINT correctly cause the game server to shut down gracefully, we're handling both events the .NET runtime raises for this.
deltanedas
commented
4 months ago you should only get sigterm if using killall dotnet etc
its also possible theres mishandling between a dotnet exec ... process and the Content.Server process thats spawned, maybe only the latter handles sigint?
dffdff2423
commented
4 months ago Press ctrl-c to kill it
That's a SIGINT, not a SIGTERM.
Oh right, my bad. It is kinda random when it happens. Like, there are sometimes long periods where it does not then it happens 10 times in a row. There is a stacktrace before it happens. I will try to past it here next time it does. I always launch the game from rider or with ./Content.Server.
dffdff2423
commented
4 months ago Okay, so it happened again and I got the stacktrace. I can grab the coredump if that is also wanted. I killed by pressing the rider stop button this time. However, this was on a fork that was a few days out of date, I still think that the issue exists upstream.
[INFO] srv: CancelKey, shutting down...
[DEBG] play_time: Saved 2 trackers
[ERRO] entity: Caught exception while raising event EntityTerminatingEvent on entity Lizard (2461/n2461, MobReptilian)
System.Collections.Generic.KeyNotFoundException: The given key 'f274edd9-d738-4dc0-8f82-71f9b5b0204e' was not present in the dictionary.
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
at Robust.Shared.Player.SharedPlayerManager.GetPlayerData(NetUserId userId) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/Player/SharedPlayerManager.Data.cs:line 16
at Content.Server.Mind.MindSystem.TryGetMind(NetUserId user, Nullable`1& mindId, MindComponent& mind) in /home/andrew/src/graphics/cosmatic-drift/Content.Server/Mind/MindSystem.cs:line 88
at Content.Shared.Mind.SharedMindSystem.TryGetMind(ICommonSession player, EntityUid& mindId, MindComponent& mind) in /home/andrew/src/graphics/cosmatic-drift/Content.Shared/Mind/SharedMindSystem.cs:line 463
at Content.Server.Administration.Systems.AdminSystem.GetPlayerInfo(SessionData data, ICommonSession session) in /home/andrew/src/graphics/cosmatic-drift/Content.Server/Administration/Systems/AdminSystem.cs:line 228
at Content.Server.Administration.Systems.AdminSystem.UpdatePlayerList(ICommonSession player) in /home/andrew/src/graphics/cosmatic-drift/Content.Server/Administration/Systems/AdminSystem.cs:line 121
at Content.Server.Administration.Systems.AdminSystem.OnPlayerDetached(PlayerDetachedEvent ev) in /home/andrew/src/graphics/cosmatic-drift/Content.Server/Administration/Systems/AdminSystem.cs:line 180
at Robust.Shared.GameObjects.EntityEventBus.<>c__DisplayClass4_0`1.<SubscribeEvent>b__0(Unit& ev) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Broadcast.cs:line 173
at Robust.Shared.GameObjects.EntityEventBus.ProcessSingleEventCore(EventSource source, Unit& unitRef, EventData subs) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Broadcast.cs:line 335
at Robust.Shared.GameObjects.EntityEventBus.RaiseLocalEventCore(EntityUid uid, Unit& unitRef, Type type, Boolean broadcast) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 224
at Robust.Shared.GameObjects.EntityEventBus.RaiseLocalEvent[TEvent](EntityUid uid, TEvent args, Boolean broadcast) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 180
at Robust.Shared.Player.SharedPlayerManager.Detach(ICommonSession session) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/Player/SharedPlayerManager.Sessions.cs:line 188
at Robust.Shared.Player.SharedPlayerManager.SetAttachedEntity(ICommonSession session, Nullable`1 uid, ICommonSession& kicked, Boolean force) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/Player/SharedPlayerManager.Sessions.cs:line 170
at Robust.Shared.Player.ISharedPlayerManager.SetAttachedEntity(ICommonSession session, Nullable`1 entity, Boolean force) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/Player/ISharedPlayerManager.cs:line 149
at Content.Server.Mind.MindSystem.TransferTo(EntityUid mindId, Nullable`1 entity, Boolean ghostCheckOverride, Boolean createGhost, MindComponent mind) in /home/andrew/src/graphics/cosmatic-drift/Content.Server/Mind/MindSystem.cs:line 259
at Content.Server.Mind.MindSystem.OnMindContainerTerminating(EntityUid uid, MindContainerComponent component, EntityTerminatingEvent& args) in /home/andrew/src/graphics/cosmatic-drift/Content.Server/Mind/MindSystem.cs:line 69
at Robust.Shared.GameObjects.EntityEventBus.<>c__DisplayClass52_0`2.<SubscribeLocalEvent>g__EventHandler|0(EntityUid uid, IComponent comp, TEvent& args) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 285
at Robust.Shared.GameObjects.EntityEventBus.<>c__DisplayClass63_0`1.<EntSubscribe>b__0(EntityUid uid, IComponent comp, Unit& ev) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 440
at Robust.Shared.GameObjects.EntityEventBus.EntDispatch(EntityUid euid, Type eventType, Unit& args) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 596
at Robust.Shared.GameObjects.EntityEventBus.RaiseLocalEventCore(EntityUid uid, Unit& unitRef, Type type, Boolean broadcast) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 220
at Robust.Shared.GameObjects.EntityEventBus.RaiseLocalEvent[TEvent](EntityUid uid, TEvent& args, Boolean broadcast) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 198
at Robust.Shared.GameObjects.EntityManager.RecursiveFlagEntityTermination(EntityUid uid, MetaDataComponent metadata, TransformComponent xform) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 527
[INFO] system.map: Grid Dev (3/n3) changed parent. Old parent: station map - Dev (4/n4). New parent: 0/n0D
[DEBG] system.npc: Sleeping mouse (416) (2329/n2329, MobMouse)
[ERRO] entity: Caught exception while trying to recursively delete child entity 'Lizard (2461/n2461, MobReptilian)' of 'Dev (3/n3)'
Robust.Shared.Utility.DebugAssertException: Exception of type 'Robust.Shared.Utility.DebugAssertException' was thrown.
at Robust.Shared.Utility.DebugTools.AssertNull(Object arg) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/Utility/DebugTools.cs:line 246
at Robust.Server.GameStates.PvsSystem.UpdatePosition(EntityUid uid, TransformComponent xform, MetaDataComponent meta, EntityUid oldParent) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/GameStates/PvsSystem.Entity.cs:line 38
at Robust.Server.GameStates.PvsSystem.OnEntityMove(MoveEvent& ev) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/GameStates/PvsSystem.Entity.cs:line 14
at Robust.Shared.GameObjects.SharedTransformSystem.RaiseMoveEvent(Entity`2 ent, EntityUid oldParent, Vector2 oldPosition, Angle oldRotation, Nullable`1 oldMap) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/Systems/SharedTransformSystem.cs:line 282
at Robust.Shared.GameObjects.SharedTransformSystem.SetCoordinates(Entity`2 entity, EntityCoordinates value, Nullable`1 rotation, Boolean unanchor, TransformComponent newParent, TransformComponent oldParent) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/Systems/SharedTransformSystem.Component.cs:line 584
at Robust.Shared.GameObjects.SharedTransformSystem.DetachEntityInternal(EntityUid uid, TransformComponent xform, MetaDataComponent meta, TransformComponent oldXform, Boolean terminating) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/Systems/SharedTransformSystem.Component.cs:line 1452
at Robust.Shared.GameObjects.SharedTransformSystem.DetachEntity(EntityUid uid, TransformComponent xform, MetaDataComponent meta, TransformComponent oldXform, Boolean terminating) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/Systems/SharedTransformSystem.Component.cs:line 1381
at Robust.Shared.GameObjects.EntityManager.RecursiveDeleteEntity(EntityUid uid, MetaDataComponent metadata, TransformComponent transform, TransformComponent parentXform) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 560
at Robust.Shared.GameObjects.EntityManager.RecursiveDeleteEntity(EntityUid uid, MetaDataComponent metadata, TransformComponent transform, TransformComponent parentXform) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 569
[INFO] system.map: Removing grid Dev (3/n3)
[ERRO] entity: Entities were spawned while flushing entities.
[FATL] unhandled: Robust.Shared.Utility.DebugAssertException: Exception of type 'Robust.Shared.Utility.DebugAssertException' was thrown.
at Robust.Shared.Utility.DebugTools.Assert(Boolean condition) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/Utility/DebugTools.cs:line 37
at Robust.Server.GameStates.PvsSystem.AfterEntityFlush() in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/GameStates/PvsSystem.DataStorage.cs:line 303
at Robust.Shared.GameObjects.EntityManager.FlushEntities() in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 713
at Robust.Shared.GameObjects.EntityManager.Cleanup() in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 236
at Robust.Server.BaseServer.Cleanup() in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/BaseServer.cs:line 660
at Robust.Server.BaseServer.FinishMainLoop() in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/BaseServer.cs:line 553
at Robust.Server.BaseServer.MainLoop() in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/BaseServer.cs:line 571
at Robust.Server.Program.ParsedMain(CommandLineArgs args, Boolean contentStart, ServerOptions options) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/Program.cs:line 76
at Robust.Server.Program.Start(String[] args, ServerOptions options, Boolean contentStart) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/Program.cs:line 44
at Robust.Server.ContentStart.Start(String[] args) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/ContentStart.cs:line 10
at Content.Server.Program.Main(String[] args) in /home/andrew/src/graphics/cosmatic-drift/Content.Server/Program.cs:line 9
Unhandled exception. Robust.Shared.Utility.DebugAssertException: Exception of type 'Robust.Shared.Utility.DebugAssertException' was thrown.
at Robust.Shared.Utility.DebugTools.Assert(Boolean condition) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/Utility/DebugTools.cs:line 37
at Robust.Server.GameStates.PvsSystem.AfterEntityFlush() in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/GameStates/PvsSystem.DataStorage.cs:line 303
at Robust.Shared.GameObjects.EntityManager.FlushEntities() in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 713
at Robust.Shared.GameObjects.EntityManager.Cleanup() in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 236
at Robust.Server.BaseServer.Cleanup() in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/BaseServer.cs:line 660
at Robust.Server.BaseServer.FinishMainLoop() in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/BaseServer.cs:line 553
at Robust.Server.BaseServer.MainLoop() in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/BaseServer.cs:line 571
at Robust.Server.Program.ParsedMain(CommandLineArgs args, Boolean contentStart, ServerOptions options) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/Program.cs:line 76
at Robust.Server.Program.Start(String[] args, ServerOptions options, Boolean contentStart) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/Program.cs:line 44
at Robust.Server.ContentStart.Start(String[] args) in /home/andrew/src/graphics/cosmatic-drift/RobustToolbox/Robust.Server/ContentStart.cs:line 10
at Content.Server.Program.Main(String[] args) in /home/andrew/src/graphics/cosmatic-drift/Content.Server/Program.cs:line 9
Process finished with exit code 134.Game bug in part but engine should be made to catch those exceptions more gracefully coming from the simulation.
dffdff2423
commented
4 months ago This was in debug mode without exception tolerance. Could that be part of it?
PJB3005
commented
4 months ago Ok I checked the relevant piece of code and this specific error doesn't appear to be caught under exception tolerance. It's only an assert but it does seem to indicate a deeper bug with shutdown logic.
So no, not just because it's a debug build.
dffdff2423
commented
4 months ago I just managed to replicate it on the latest master of wizden.
[INFO] srv: CancelKey, shutting down...
[DEBG] play_time: Saved 2 trackers
[ERRO] entity: Caught exception while raising event EntityTerminatingEvent on entity Armando Draudy (2502/n2502, MobSlimePerson)
System.Collections.Generic.KeyNotFoundException: The given key '65993eb0-456a-48e4-92fe-82eb05f11ec5' was not present in the dictionary.
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
at Robust.Shared.Player.SharedPlayerManager.GetPlayerData(NetUserId userId) in /home/andrew/src/graphics/space-station-14/RobustToolbox/Robust.Shared/Player/SharedPlayerManager.Data.cs:line 16
at Content.Server.Mind.MindSystem.TryGetMind(NetUserId user, Nullable`1& mindId, MindComponent& mind) in /home/andrew/src/graphics/space-station-14/Content.Server/Mind/MindSystem.cs:line 88
at Content.Shared.Mind.SharedMindSystem.TryGetMind(ICommonSession player, EntityUid& mindId, MindComponent& mind) in /home/andrew/src/graphics/space-station-14/Content.Shared/Mind/SharedMindSystem.cs:line 463
at Content.Server.Administration.Systems.AdminSystem.GetPlayerInfo(SessionData data, ICommonSession session) in /home/andrew/src/graphics/space-station-14/Content.Server/Administration/Systems/AdminSystem.cs:line 222
at Content.Server.Administration.Systems.AdminSystem.UpdatePlayerList(ICommonSession player) in /home/andrew/src/graphics/space-station-14/Content.Server/Administration/Systems/AdminSystem.cs:line 115
at Content.Server.Administration.Systems.AdminSystem.OnPlayerDetached(PlayerDetachedEvent ev) in /home/andrew/src/graphics/space-station-14/Content.Server/Administration/Systems/AdminSystem.cs:line 174
at Robust.Shared.GameObjects.EntityEventBus.<>c__DisplayClass4_0`1.
This description is not an accurate description of the issue. See PJB's comment below.
Description
I am not 100% sure if this is a SS14 issue or a RobustToolbox issue.
When you kill the SS14 server with a SIGTERM (ctrl-c) on linux it coredumps sometimes. SS14 should install a signal handler that executes the shutdown command for a graceful shutdown. I think it is possible to lose data due to this if the database is busy which is not the expected behavior of SIGTERM.
Reproduction
Screenshots
N/A
Additional context
I don't think telling people to just not use ctrl-c is a good idea because of the possibility of dataloss. At least for me, it is just in my muscle memory to kill processes with ctrl-c. Also, my coredumps folder was over 2GiB from ss14 related coredumps.