Server randomly hangs

[iagrib]

My server sometimes randomly freezes. It doesn't crash: no crash logs are created, no errors are thrown, console just stops producing any output and the server must be manually restarted.

It happens when game.CleanUpMap() is being called, but I didn't find any other conditions that might cause this.

I've found numerous threads about same or similar issue on Facepunch forums:

https://facepunch.com/showthread.php?t=1286685

https://facepunch.com/showthread.php?t=1415912

https://facepunch.com/showthread.php?t=1359264

https://facepunch.com/showthread.php?t=1320466

https://facepunch.com/showthread.php?t=1572316

I couldn't find any similarities between them but that most (but not all) of people experiencing this are running TTT servers on Linux machines, including me. So I think it's an issue related to gmod/srcds itself.

[robotboy655]

Have you tried the simplest thing of not running any mods before posting here?

[iagrib]

I tried disabling my addons one by one, as well as inspecting their code, but I couldn't find anything that could cause this.

[Kefta]

Does this happen with no addons? On sandbox?

[iagrib]

I can't say for sure, it's quite hard to test it because of seeming randomness of this crash - it would require many hours of playtesting

[Kefta]

Sounds like a duplicate of https://github.com/Facepunch/garrysmod-issues/issues/3211

[BlackCetha]

Thread revive! Instead of crashing like in #3211 my server now freezes with 100% cpu usage and no other activity. I attached GDB to it and got the following backtrace:

#0  0xf37583ef in CUtlMultiList<void*, unsigned short>::Remove(unsigned short, unsigned short) () from target:/opt/gmod/bin/vphysics_srv.so
#1  0xf375ce1e in CObjectPairHash::RemoveAllPairsForObject(void*) () from target:/opt/gmod/bin/vphysics_srv.so
#2  0xf2768921 in PhysDestroyObject(IPhysicsObject*, CBaseEntity*) () from target:/opt/gmod/garrysmod/bin/server_srv.so
#3  0xf2648389 in CBaseEntity::VPhysicsDestroyObject() () from target:/opt/gmod/garrysmod/bin/server_srv.so
#4  0xf29c230f in CBaseEntity::UpdateOnRemove() () from target:/opt/gmod/garrysmod/bin/server_srv.so
#5  0xf2f60cec in UTIL_Remove(IServerNetworkable*) () from target:/opt/gmod/garrysmod/bin/server_srv.so
#6  0xf269e9ab in CGMODRules::CleanUpMap(CLuaObject&) () from target:/opt/gmod/garrysmod/bin/server_srv.so
#7  0xf27c6c36 in real__game__CleanUpMap(ILuaInterface*) () from target:/opt/gmod/garrysmod/bin/server_srv.so
#8  0xef0f8249 in lj_BC_FUNCC () from target:/opt/gmod/garrysmod/bin/lua_shared_srv.so
#9  0xef0eb6c8 in lua_pcall () from target:/opt/gmod/garrysmod/bin/lua_shared_srv.so
#10 0xef0b570c in CLuaInterface::PCall(int, int, int) () from target:/opt/gmod/garrysmod/bin/lua_shared_srv.so
#11 0xef0bef86 in CLuaInterface::CallFunctionProtected(int, int, bool) () from target:/opt/gmod/garrysmod/bin/lua_shared_srv.so
#12 0xf27da426 in GarrysMod::Lua::Libraries::Timer::CallTimerFunction(int, std::string const&, std::string const&) () from target:/opt/gmod/garrysmod/bin/server_srv.so
#13 0xf27da87b in GarrysMod::Lua::Libraries::Timer::DoAdvancedTimers() () from target:/opt/gmod/garrysmod/bin/server_srv.so
#14 0xf27da90f in GarrysMod::Lua::Libraries::Timer::Cycle() () from target:/opt/gmod/garrysmod/bin/server_srv.so
#15 0xf27a97fa in CGarrysMod::Think() () from target:/opt/gmod/garrysmod/bin/server_srv.so
#16 0xf26a0573 in CLuaGameSystem::FrameUpdatePreEntityThink() () from target:/opt/gmod/garrysmod/bin/server_srv.so
#17 0xf273f9dd in InvokePerFrameMethod(void (IGameSystemPerFrame::*)(), char const*) [clone .isra.20] [clone .constprop.46] () from target:/opt/gmod/garrysmod/bin/server_srv.so
#18 0xf27402d2 in IGameSystem::FrameUpdatePreEntityThinkAllSystems() () from target:/opt/gmod/garrysmod/bin/server_srv.so
#19 0xf2a9c1bb in CServerGameDLL::GameFrame(bool) () from target:/opt/gmod/garrysmod/bin/server_srv.so
#20 0xf6436ea8 in SV_Think(bool) () from target:/opt/gmod/bin/engine_srv.so
#21 0xf6437fce in SV_Frame(bool) () from target:/opt/gmod/bin/engine_srv.so
#22 0xf63b5d2e in _Host_RunFrame_Server(bool) () from target:/opt/gmod/bin/engine_srv.so
#23 0xf63b6684 in _Host_RunFrame(float) () from target:/opt/gmod/bin/engine_srv.so
#24 0xf63c2ed8 in CHostState::State_Run(float) () from target:/opt/gmod/bin/engine_srv.so
#25 0xf63c31b6 in CHostState::FrameUpdate(float) () from target:/opt/gmod/bin/engine_srv.so
#26 0xf63c31fd in HostState_Frame(float) () from target:/opt/gmod/bin/engine_srv.so
#27 0xf644d1d8 in CEngine::Frame() () from target:/opt/gmod/bin/engine_srv.so
#28 0xf644ad86 in CDedicatedServerAPI::RunFrame() () from target:/opt/gmod/bin/engine_srv.so
#29 0xf784673a in ?? ()
#30 0xf644ae2d in CModAppSystemGroup::Main() () from target:/opt/gmod/bin/engine_srv.so
#31 0xf64a0b68 in CAppSystemGroup::Run() () from target:/opt/gmod/bin/engine_srv.so
#32 0xf644b13d in CDedicatedServerAPI::ModInit(ModInfo_t&) () from target:/opt/gmod/bin/engine_srv.so
#33 0xf78463f8 in ?? ()
#34 0xf78a3bb8 in ?? ()
#35 0xf78a3bb8 in ?? ()
#36 0xf77f227a in ?? ()
#37 0x08048aab in main ()

I printed the backtrace multiple times and it seems like it is stuck in CObjectPairHash::RemoveAllPairsForObject(void*) from vphysics_srv.so. At the time I took this backtrace, it had been going for nine hours.

[livingflore]

+1

#0  0xf3682f36 in CObjectPairHash::RemoveObjectPair(void*, void*) () from /home/nonstop/ttt/bin/vphysics_srv.so
#1  0xf2630b09 in PhysEnableEntityCollisions(CBaseEntity*, CBaseEntity*) () from /home/nonstop/ttt/garrysmod/bin/server_srv.so
#2  0xf2d55bbc in CPhysicsEntitySolver::UpdateOnRemove() () from /home/nonstop/ttt/garrysmod/bin/server_srv.so
#3  0xf2e5283c in UTIL_Remove(IServerNetworkable*) () from /home/nonstop/ttt/garrysmod/bin/server_srv.so
#4  0xf2d505fd in CBaseEntity::PhysicsDispatchThink(void (CBaseEntity::*)()) () from /home/nonstop/ttt/garrysmod/bin/server_srv.so
#5  0xf2616984 in CBaseEntity::PhysicsRunSpecificThink(int, void (CBaseEntity::*)()) () from /home/nonstop/ttt/garrysmod/bin/server_srv.so
#6  0xf2616ee2 in CBaseEntity::PhysicsRunThink(CBaseEntity::thinkmethods_t) [clone .part.156] () from /home/nonstop/ttt/garrysmod/bin/server_srv.so
#7  0xf2d53ae0 in Physics_SimulateEntity(CBaseEntity*) () from /home/nonstop/ttt/garrysmod/bin/server_srv.so
#8  0xf2d53d41 in Physics_RunThinkFunctions(bool) () from /home/nonstop/ttt/garrysmod/bin/server_srv.so
#9  0xf2975b04 in CServerGameDLL::GameFrame(bool) () from /home/nonstop/ttt/garrysmod/bin/server_srv.so
#10 0xf636a538 in SV_Think(bool) () from /home/nonstop/ttt/bin/engine_srv.so
#11 0xf636b6b3 in SV_Frame(bool) () from /home/nonstop/ttt/bin/engine_srv.so
#12 0xf62e3d2e in _Host_RunFrame_Server(bool) () from /home/nonstop/ttt/bin/engine_srv.so
#13 0xf62e488c in _Host_RunFrame(float) () from /home/nonstop/ttt/bin/engine_srv.so
#14 0xf62f1618 in CHostState::State_Run(float) () from /home/nonstop/ttt/bin/engine_srv.so
#15 0xf62f18f6 in CHostState::FrameUpdate(float) () from /home/nonstop/ttt/bin/engine_srv.so
#16 0xf62f193d in HostState_Frame(float) () from /home/nonstop/ttt/bin/engine_srv.so
#17 0xf6381c19 in CEngine::Frame() () from /home/nonstop/ttt/bin/engine_srv.so
#18 0xf637f4e6 in CDedicatedServerAPI::RunFrame() () from /home/nonstop/ttt/bin/engine_srv.so
#19 0xf774aa5a in ?? ()
#20 0xf637f58d in CModAppSystemGroup::Main() () from /home/nonstop/ttt/bin/engine_srv.so
#21 0xf63d5d98 in CAppSystemGroup::Run() () from /home/nonstop/ttt/bin/engine_srv.so
#22 0xf637f89d in CDedicatedServerAPI::ModInit(ModInfo_t&) () from /home/nonstop/ttt/bin/engine_srv.so
#23 0xf774a718 in ?? ()
#24 0xf77a8e78 in ?? ()
#25 0xf77a8e78 in ?? ()
#26 0xf76f5b4a in ?? ()
#27 0x08048aab in main ()

upd: confirmed with physgun crash tool

/proc/1/fd/4(+0x2db1b)[0xf797eb1b]
/usr/lib32/libstdc++.so.6(+0xb26cd)[0xf7b9a6cd]
/lib/i386-linux-gnu/libpthread.so.0(+0x70b4)[0xf7ea20b4]
/lib/i386-linux-gnu/libc.so.6(clone+0x66)[0xf7db2386]

Lua Stack:
[C][+0] [C] in CleanUpMap
[L][+1] addons/tttdamagelogs/lua/damagelogs/shared/sync.lua:42 in field CleanUpMap Line 37 -> 44
[L][+2] gamemodes/terrortown/gamemode/init.lua:384 in upvalue CleanUp Line 376 -> 397
[L][+3] gamemodes/terrortown/gamemode/init.lua:467 in anonymous function Line 450 -> 519

[wrefgtzweve]

Just had this happen too on TTT after game.CleanUpMap, stuff like EntityRemoved got number values and it eventually crashed with a lua panic.

[wrefgtzweve]

I think the issue might be related to damage logs and specifically this commit, https://github.com/BadgerCode/tttdamagelogs/commit/843bdc4649047c7e20117c1d32a119a689e99f51

The EFlag EFL_KEEP_ON_RECREATE_ENTITIES might be causing this but im not fully sure yet as its hard to replicate for me.

[yuuto-ichika]

tttpatch.zip The issue stems from CleanUpMap attempting to clean up an entity that has been :Remove()d. Why is it doing this? No idea, but there is a solution! Disable :Remove()! Surely this will have no unintended side effects! This addon detours Entity:Remove(), disabling the default functionality. Of course, it may negatively affect gameplay if nothing can be removed. By making it intangible and invisible, an entity is effectively removed, even if it isn't actually removed. So when game.CleanUpMap() makes its rounds, it will never come across an already deleted entity. Use at your own risk, but for what it's worth, our server has had minimal issues (and almost no crashes!) for a couple months.

[wrefgtzweve]

@draughtnyan Did you have the crash issue before this fix? if so, are you using badgers tttdamagelogs? Just wondering

[yuuto-ichika]

@draughtnyan Did you have the crash issue before this fix? if so, are you using badgers tttdamagelogs? Just wondering

Yes and yes. We were using an older version of damagelogs until recently though. The crashes had been an issue since at least April of this year by my observation, but I have been told they were an issue even before then.

[livingflore]

tttpatch.zip The issue stems from CleanUpMap attempting to clean up an entity that has been :Remove()d. Why is it doing this? No idea, but there is a solution! Disable :Remove()! Surely this will have no unintended side effects! This addon detours Entity:Remove(), disabling the default functionality. Of course, it may negatively affect gameplay if nothing can be removed. By making it intangible and invisible, an entity is effectively removed, even if it isn't actually removed. So when game.CleanUpMap() makes its rounds, it will never come across an already deleted entity. Use at your own risk, but for what it's worth, our server has had minimal issues (and almost no crashes!) for a couple months.

unfortunately didn't help i'm using tommy228 damage logs

[yuuto-ichika]

I'm pretty sure the only reason that the damagelogs plugin appears in the crash stack trace is because it overrides CleanUpMap to exclude the sync entity, I don't think it has anything to do with the crash.

[yuuto-ichika]

unfortunately didn't help

fix_cleanup.lua.txt swep_remove_log.lua.txt Can you try adding these to shared too (remove the .txt)? They theoretically shouldn't do anything, but we did have issues for a short period of time when I deleted these files during some old file cleaning. I put the files back, but still regarded it as a coincidence, and figured they really did do nothing. But they might do something.

[livingflore]

unfortunately didn't help

fix_cleanup.lua.txt swep_remove_log.lua.txt Can you try adding these to shared too (remove the .txt)? They theoretically shouldn't do anything, but we did have issues for a short period of time when I deleted these files during some old file cleaning. I put the files back, but still regarded it as a coincidence, and figured they really did do nothing. But they might do something.

it seems like it didn't load properly in first place since i renamed tttpatch folder to [fix]_EntityRemove, after i renamed it back and added those lua files - i can't trigger lua panic at all! appreciate your help a lot

[yuuto-ichika]

epic, so just to be sure, those two files are irrelevant and tttpatch.zip is the only one that is needed?

[livingflore]

epic, so just to be sure, those two files are irrelevant and tttpatch.zip is the only one that is needed?

not actually sure, will recheck and come back to you

[livingflore]

epic, so just to be sure, those two files are irrelevant and tttpatch.zip is the only one that is needed?

tested again w/o those 2 files, can't trigger lua panic as well, though i experience some minor glitches like when you place traitor turret from t shop, for some reason it spawns twice - actual turret and its broken prop that you can't interact with