Server crashes after a few hours of runtime.

[KodakDolph]

Opened up a server today and have been experiencing some lag/crashing errors that I can't seem to find the root cause of. It kinda makes sense given that we have been seeing an average of ~50 players, I've noticed that the issues/ping worsen when players are killing each other in PVP. eventually, after a few hours, it will just crash with minimal logging errors. The server itself doesn't actually turn off but everyone is kicked, no one can rejoin and all logging stops.

I have found this error in log spammed many times

[20:04:52 ERR][SDG.Unturned] Steamworks.CSteamID SDG.Unturned.SteamPlayerID.get_steamID() [20:04:52 ERR][SDG.Unturned] at (wrapper delegate-invoke) .invoke_void_PlayerLife_EDeathCause_ELimb_CSteamID(SDG.Un turned.PlayerLife,SDG.Unturned.EDeathCause,SDG.Unturned.ELimb,Steamworks.CSteamID) at SDG.Unturned.PlayerLife.broadcastPlayerDied (SDG.Unturned.PlayerLife sender, SDG.Unturned.EDeathCause cause, SDG.Unturne d.ELimb limb, Steamworks.CSteamID instigator) [0x00009] in <1701fb1dfda14db29091b5f94d65785a>:0

I have tried unloading several plugins that I thought may be associated with deaths as that error suggests but I haven't really found anything or had any luck.

If there is anything that I can provide (logs, etc) please direct me to them.

[KodakDolph]

Update: have found this error being spawned literally nonstop in Server_server.log:

[2022-06-26 23:38:10] MissingMethodException: void SDG.Unturned.StructureManager.damage(UnityEngine.Transform,UnityEngine.Vector3,single,single,bool,Steamworks.CSteamID,SDG.Unturned.EDamageOrigin)

[GazziFX]

Im sure its caused by outdated plugin but cant define it because you provided just small piece, there should be longer stacktrace

[KodakDolph]

Im sure its caused by outdated plugin but cant define it because you provided just small piece, there should be longer stacktrace

Where exactly could I find the full stacktrace?

[KodakDolph]

Okay, I think I may have found what you are talking about.

[2022-06-27 12:55:58] Plugin raised an exception from onPlayerDied: [2022-06-27 12:55:58] Object reference not set to an instance of an object [2022-06-27 12:55:58] at Rocket.Unturned.Player.UnturnedPlayer.TriggerEffect (System.UInt16 effectID) [0x00008] in :0 at KodakCore.Main.OnDeath (SDG.Unturned.PlayerLife sender, SDG.Unturned.EDeathCause cause, SDG.Unturned.ELimb limb, Steamworks.CSteamID instigator) [0x00069] in :0 at (wrapper delegate-invoke) .invoke_void_PlayerLife_EDeathCause_ELimb_CSteamID(SDG.Unturned.PlayerLife,SDG.Unturned.EDeathCause,SDG.Unturned.ELimb,Steamworks.CSteamID) at SDG.Unturned.PlayerLife.broadcastPlayerDied (SDG.Unturned.PlayerLife sender, SDG.Unturned.EDeathCause cause, SDG.Unturned.ELimb limb, Steamworks.CSteamID instigator) [0x00009] in <1701fb1dfda14db29091b5f94d65785a>:0

"KodakCore" is a private plugin I use. Do you think this error is the reason for my server's crashing? At one point i /rocket unloaded this plugin and it still crashed like 20 minutes later.

[KodakDolph]

Tried deleting that KodakCore plugin and the same thing happened. The server ran perfectly smooth for about an hour then just disconnected everyone. Logging just stops the exact second it all starts

[SDGNelson]

It's odd that this would come up all of a sudden because I do not think those have changed recently, not to mention it has been a while since the last update now. When you mention the server ran normally for about an hour were there any other errors in the logs before it started getting stuck?

[KodakDolph]

None that I could find other than the ones I've listed, but maybe I didn't check all of the right log files.

Knock on wood, but about 5 hours ago I deleted a few plugins and the server hasn't done the kick in over 5 hours (which is a record). I will update things here

[KodakDolph]

Sadly it happened again :(

Where can I look to provide logging or to better find the root of the issue?

[KodakDolph]

[2022-06-27 17:20:47] HTTP (SteamCache,276) - cache3-lax1.steamcontent.com (162.254.195.13:80 / 162.254.195.13:80, host: cache3-lax1.steamcontent.com): Closing connection [2022-06-27 17:20:47] HTTP (SteamCache,8) - cache3-sea1.steamcontent.com (205.196.6.171:80 / 205.196.6.171:80, host: cache3-sea1.steamcontent.com): Closing connection [2022-06-27 17:20:51] HTTP (SteamCache,1) - cache1-sea1.steamcontent.com (205.196.6.165:443 / 205.196.6.165:443, host: cache1-sea1.steamcontent.com): Closing connection

found this in content_log.txt

could it be the issue?

[KodakDolph]

Was able to find this error on the backend. What does it mean?

[GazziFX]

Can you upload file from given path?

[KodakDolph]

Can you upload file from given path?

You mean like the crash report?

[GazziFX]

yes this one

[KodakDolph]

yes this one

Crash_2022-06-28_134234284.zip

I believe this is what you're asking for. Forgive me for being a little clueless

[GazziFX]

Oh it doesnt contain any logs?

[KodakDolph]

Oh it doesnt contain any logs?

nope. would they typically be in there?

[GazziFX]

I dont know, but if it happens again can you copy full text of error, especially this stacktrace, it can contain name of method caused crash

[SDGNelson]

Thanks for the crash zip! It contains crash.dmp which has all the info needed to get the call stack:

UnityPlayer.dll!physx::Gu::AABBTreeRaycast<0,physx::Sq::AABBTree,physx::Sq::AABBTreeRuntimeNode,physx::Sq::PrunerPayload,physx::Sq::PrunerCallback>::operator()(const physx::Sq::PrunerPayload * objects, const physx::PxBounds3 * boxes, const physx::Sq::AABBTree & tree, const physx::PxVec3 & origin, const physx::PxVec3 & unitDir, float & maxDist, const physx::PxVec3 & inflation, physx::Sq::PrunerCallback & pcb) Line 225    C++
UnityPlayer.dll!physx::Sq::AABBPruner::raycast(const physx::PxVec3 & origin, const physx::PxVec3 & unitDir, float & inOutDistance, physx::Sq::PrunerCallback & pcb) Line 350    C++
UnityPlayer.dll!physx::NpSceneQueries::multiQuery<physx::PxRaycastHit>(const physx::MultiQueryInput & input, physx::PxHitCallback<physx::PxRaycastHit> & hits, physx::PxFlags<enum physx::PxHitFlag::Enum,unsigned short> hitFlags, const physx::PxQueryCache * cache, const physx::PxQueryFilterData & filterData, physx::PxQueryFilterCallback * filterCall, physx::BatchQueryFilterData * bfd) Line 772  C++
UnityPlayer.dll!physx::NpBatchQuery::execute() Line 394 C++
UnityPlayer.dll!physx::PxVehicleUpdate::suspensionRaycasts(physx::PxBatchQuery * batchQuery, const unsigned int numVehicles, physx::PxVehicleWheels * * vehicles, const unsigned int numSceneQueryResults, physx::PxBatchQueryResult<physx::PxRaycastHit> * sceneQueryResults, const bool * vehiclesToRaycast) Line 7455    C++
UnityPlayer.dll!VehiclesManager::VehicleConfig::UpdateVehicles(const float deltaTime) Line 374  C++
UnityPlayer.dll!VehiclesManager::UpdateVehicles(const int physicsSceneHandle, const float deltaTime) Line 483   C++
UnityPlayer.dll!PhysicsManager::Simulate(int sceneHandle, float dt) Line 817    C++
[Inline Frame] UnityPlayer.dll!PhysicsManager::FixedUpdate() Line 724   C++
UnityPlayer.dll!`PhysicsManager::InitializeClass'::`2'::FixedUpdatePhysicsFixedUpdateRegistrator::Forward() Line 434    C++
UnityPlayer.dll!ExecutePlayerLoop(NativePlayerLoopSystem * system) Line 384 C++
UnityPlayer.dll!ExecutePlayerLoop(NativePlayerLoopSystem * system) Line 408 C++
UnityPlayer.dll!PlayerLoop() Line 514   C++
UnityPlayer.dll!PerformMainLoop() Line 324  C++
UnityPlayer.dll!MainMessageLoop() Line 1202 C++
UnityPlayer.dll!UnityMainImpl(HINSTANCE__ * hInst, HINSTANCE__ * hPrev, wchar_t * szCmdLine, int nCmdShow) Line 1703    C++
UnityPlayer.dll!UnityMain(HINSTANCE__ * hInst, HINSTANCE__ * hPrev, wchar_t * szCmdLine, int nCmdShow) Line 1758    C++

Unfortunately the crash is happening within the physics system which is entirely out of our control. That being said, it is odd if this crash is frequent on your server but not others. If you try running a vanilla map without any workshop files installed does the server crash? I ask because maybe there is something wonky in the physics configuration of some custom content causing the crash.

[KodakDolph]

Thanks for the crash zip! It contains crash.dmp which has all the info needed to get the call stack:

UnityPlayer.dll!physx::Gu::AABBTreeRaycast<0,physx::Sq::AABBTree,physx::Sq::AABBTreeRuntimeNode,physx::Sq::PrunerPayload,physx::Sq::PrunerCallback>::operator()(const physx::Sq::PrunerPayload * objects, const physx::PxBounds3 * boxes, const physx::Sq::AABBTree & tree, const physx::PxVec3 & origin, const physx::PxVec3 & unitDir, float & maxDist, const physx::PxVec3 & inflation, physx::Sq::PrunerCallback & pcb) Line 225  C++
UnityPlayer.dll!physx::Sq::AABBPruner::raycast(const physx::PxVec3 & origin, const physx::PxVec3 & unitDir, float & inOutDistance, physx::Sq::PrunerCallback & pcb) Line 350  C++
UnityPlayer.dll!physx::NpSceneQueries::multiQuery<physx::PxRaycastHit>(const physx::MultiQueryInput & input, physx::PxHitCallback<physx::PxRaycastHit> & hits, physx::PxFlags<enum physx::PxHitFlag::Enum,unsigned short> hitFlags, const physx::PxQueryCache * cache, const physx::PxQueryFilterData & filterData, physx::PxQueryFilterCallback * filterCall, physx::BatchQueryFilterData * bfd) Line 772    C++
UnityPlayer.dll!physx::NpBatchQuery::execute() Line 394   C++
UnityPlayer.dll!physx::PxVehicleUpdate::suspensionRaycasts(physx::PxBatchQuery * batchQuery, const unsigned int numVehicles, physx::PxVehicleWheels * * vehicles, const unsigned int numSceneQueryResults, physx::PxBatchQueryResult<physx::PxRaycastHit> * sceneQueryResults, const bool * vehiclesToRaycast) Line 7455  C++
UnityPlayer.dll!VehiclesManager::VehicleConfig::UpdateVehicles(const float deltaTime) Line 374    C++
UnityPlayer.dll!VehiclesManager::UpdateVehicles(const int physicsSceneHandle, const float deltaTime) Line 483 C++
UnityPlayer.dll!PhysicsManager::Simulate(int sceneHandle, float dt) Line 817  C++
[Inline Frame] UnityPlayer.dll!PhysicsManager::FixedUpdate() Line 724 C++
UnityPlayer.dll!`PhysicsManager::InitializeClass'::`2'::FixedUpdatePhysicsFixedUpdateRegistrator::Forward() Line 434  C++
UnityPlayer.dll!ExecutePlayerLoop(NativePlayerLoopSystem * system) Line 384   C++
UnityPlayer.dll!ExecutePlayerLoop(NativePlayerLoopSystem * system) Line 408   C++
UnityPlayer.dll!PlayerLoop() Line 514 C++
UnityPlayer.dll!PerformMainLoop() Line 324    C++
UnityPlayer.dll!MainMessageLoop() Line 1202   C++
UnityPlayer.dll!UnityMainImpl(HINSTANCE__ * hInst, HINSTANCE__ * hPrev, wchar_t * szCmdLine, int nCmdShow) Line 1703  C++
UnityPlayer.dll!UnityMain(HINSTANCE__ * hInst, HINSTANCE__ * hPrev, wchar_t * szCmdLine, int nCmdShow) Line 1758  C++

Unfortunately the crash is happening within the physics system which is entirely out of our control. That being said, it is odd if this crash is frequent on your server but not others. If you try running a vanilla map without any workshop files installed does the server crash? I ask because maybe there is something wonky in the physics configuration of some custom content causing the crash.

Hey thanks for the hand Nelson. By this are you implying that it’s more likely a workshop file issue and NOT a rocketmod/openmod plugin? I do have a few privately commissioned workshop files that may be at fault here. I’ll remove some of them and see what happens.

[KodakDolph]

I also mentions of “ Vehicle” in that error perhaps it’s workshop vehicles ?

[SDGNelson]

For this crash I think it is unlikely to be plugin related because plugins would probably not be modifying physics/collision data directly. It does mention vehicles because the crash is happening in the physics engine's wheeled vehicle physics code, but if it does turn out to be bad physics/collision data then it could be from the wheels or something else. Obviously ideally it should not be crashing in there at all, but the reason I think it might be worth trying without modded content is that we have not seen this crash on other servers.

[KodakDolph]

[21:31:30 INF][OpenMod.Core.Commands.CommandExecutor] Actor player/[Hades] MegaSalamance (76561198088735492) has executed command "kit ares" attempted to call a pure virtual function Stack trace: 0x00007ff936725f5b (UnityPlayer) UnityMain 0x00007ff936725a1c (UnityPlayer) UnityMain 0x00007ff935dd8193 (UnityPlayer) UnityMain 0x00007ff936140137 (UnityPlayer) UnityMain 0x0000022fb3fe9d19 (Mono JIT Code) (wrapper managed-to-native) UnityEngine.CharacterController:Move_Injected (UnityEngine.CharacterController,UnityEngine.Vector3&) 0x0000022fb3fe9c5b (Mono JIT Code) UnityEngine.CharacterController:Move (UnityEngine.Vector3) 0x0000022fb621a403 (Mono JIT Code) AIPath:move (single) 0x0000022fb3fe9253 (Mono JIT Code) SDG.Unturned.Zombie:tick () 0x0000022e7845c0e3 (Mono JIT Code) SDG.Unturned.ZombieManager:Update () 0x0000022e2ba01d20 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_voidthis (object,intptr,intptr,intptr) 0x00007ff9439cf1e0 (mono-2.0-bdwgc) mono_get_runtime_build_info 0x00007ff943952ac2 (mono-2.0-bdwgc) mono_perfcounters_init 0x00007ff94395bb1f (mono-2.0-bdwgc) mono_runtime_invoke 0x00007ff935e7dadd (UnityPlayer) UnityMain 0x00007ff935e7abd2 (UnityPlayer) UnityMain 0x00007ff935e62473 (UnityPlayer) UnityMain 0x00007ff935e6254f (UnityPlayer) UnityMain 0x00007ff935bd1e00 (UnityPlayer) UnityMain 0x00007ff935d0839a (UnityPlayer) UnityMain 0x00007ff935d08440 (UnityPlayer) UnityMain 0x00007ff935d0c6d8 (UnityPlayer) UnityMain 0x00007ff935acc95a (UnityPlayer) 0x00007ff935acae2b (UnityPlayer) 0x00007ff935acfdfb (UnityPlayer) 0x00007ff935ad0f1b (UnityPlayer) UnityMain 0x00007ff7ae1a11f2 (Unturned) 0x00007ff98b627974 (KERNEL32) BaseThreadInitThunk 0x00007ff98e21a2f1 (ntdll) RtlUserThreadStart

I also found this ;c

[SDGNelson]

That crash might be the same one since it is within physics code, but if you can attach the crash.dmp file from the latest Crashes folder again I can double-check.

[KodakDolph]

That crash might be the same one since it is within physics code, but if you can attach the crash.dmp file from the latest Crashes folder again I can double-check.

Crash_2022-06-30_013133419.zip Here it is

[SDGNelson]

It looks like the call stack here was:

>   UnityPlayer.dll!winutils::CrashHandler::DefaultPureVirtualCallHandler() Line 48 C++
    [External Code] 
    UnityPlayer.dll!physx::Cct::Controller::move(physx::Cct::SweptVolume & volume, const physx::PxVec3 & originalDisp, float minDist, float elapsedTime, const physx::PxControllerFilters & filters, const physx::PxObstacleContext * obstacleContext, bool constrainedClimbingMode) Line 2345  C++
    UnityPlayer.dll!physx::Cct::CapsuleController::move(const physx::PxVec3 & disp, float minDist, float elapsedTime, const physx::PxControllerFilters & filters, const physx::PxObstacleContext * obstacles) Line 2557 C++
    UnityPlayer.dll!CharacterController::Move(const Vector3f & movement) Line 306   C++

Was this when the server was running with only vanilla maps and content?

[KodakDolph]

It looks like the call stack here was:

> UnityPlayer.dll!winutils::CrashHandler::DefaultPureVirtualCallHandler() Line 48 C++
  [External Code] 
  UnityPlayer.dll!physx::Cct::Controller::move(physx::Cct::SweptVolume & volume, const physx::PxVec3 & originalDisp, float minDist, float elapsedTime, const physx::PxControllerFilters & filters, const physx::PxObstacleContext * obstacleContext, bool constrainedClimbingMode) Line 2345  C++
  UnityPlayer.dll!physx::Cct::CapsuleController::move(const physx::PxVec3 & disp, float minDist, float elapsedTime, const physx::PxControllerFilters & filters, const physx::PxObstacleContext * obstacles) Line 2557 C++
  UnityPlayer.dll!CharacterController::Move(const Vector3f & movement) Line 306   C++

Was this when the server was running with only vanilla maps and content?

This was not. Admittingly my server depends so largely upon many of the workshops that it would be practically unplayable with no workshop content loaded, so I haven't gotten around to organizing that test yet.

I do have a custom plugin that can edit player gravity, height, and speed when specified with certain clothing pieces. It was made prior to the Inertia update, perhaps that is posing an issue. Do you think this could be a possible cause?

[KodakDolph]

Disregard the previous comment. Ran the server without those plugins and still crashed, so I’ll have to check out the disabling of some workshops.

[KodakDolph]

Just wanted to update.

Found the cause! It was an OpenMod home plugin. Sucks it was right under my eyes this whole time but just glad it’s fixed lol!

beware if using JZ home plugin.

[SDGNelson]

Glad to hear you got it working! So it has not been crashing anymore after disabling that plugin? It is very surprising that a plugin could be causing a low-level physics engine crash like that. Is the source code available to take a look at?

[Johnanater]

Glad to hear you got it working! So it has not been crashing anymore after disabling that plugin? It is very surprising that a plugin could be causing a low-level physics engine crash like that. Is the source code available to take a look at?

I think he's talking about this plugin: https://www.nuget.org/packages/JH.EzHomeCmd

I didn't find any source, but I decompiled it and I might have found the issue. In the CommandHome class, he calls teleportToBed() while on a different thread. I'm not sure this would cause a low-level physics crash, but it definitely wouldn't help.

[rube200]

Yes it can be that, since teleport to bed check for collision above the bed(to check if there is size for player or bed is blocked) it calls unity functions, and those don't like to be called in a different thread causing server to crash. I had a lot of those in past

[SDGNelson]

Thanks @Johnanater! Sigh, I hoped most plugins knew not to do that these days. I will add some more assertIsGameThread calls around some methods commonly used by plugins.