[L4D2][Linux] Segfault crashes when game being created in server [CSteamWorksGameStatsUploader::GetInterface()]

[rdroimfd]

I've had this issue for a long time, and is very annoying for me since I always play on my own Linux hosted server. I've been dealing with this for about 2 years (I had a server then, didn't for about a year, and now did a new install and have the same problem). I've had this issue hosting via a computer of my own, and two different VPS hosting companies with Debian.

The issue is that the server crashes with a segmentation fault on any subsequent games created. What I mean is that if a game is played on the server, everyone leaves, and a new game is created, it will crash with a segmentation fault. If I manually stop and restart the l4d2 server after each game played, it never crashes.

The segmentation faults seem to happen late into loads, right around when Round_Start happens and when the loading bar is at 100% and about to start cinematics.

I've attached the debug.log it created, however it won't let me upload the coredump file named 'core' that I got it says it's unsupported, I uploaded it here

Server details:

Distro: Debian GNU/Linux 9.4 (stretch) Arch: x86_64 Kernel: 4.9.0-6-amd64 tmux: tmux 2.3 GLIBC: 2.24

[rdroimfd]

If I run the server through gdb following this runtime debugging page, I got more info by typing bt after the segmentation fault

Thread 1 "srcds_linux" received signal SIGSEGV, Segmentation fault.
0xedf27a2e in CSteamWorksGameStatsUploader::GetInterface() ()
   from /home/l4d2server/serverfiles/left4dead2/bin/server_srv.so
(gdb) bt
#0  0xedf27a2e in CSteamWorksGameStatsUploader::GetInterface() ()
   from /home/l4d2server/serverfiles/left4dead2/bin/server_srv.so
#1  0xedf284db in CSteamWorksGameStatsUploader::InitializeSystem() ()
   from /home/l4d2server/serverfiles/left4dead2/bin/server_srv.so
#2  0xedf28eee in CSteamWorksGameStatsUploader::FireGameEvent(IGameEvent*) ()
   from /home/l4d2server/serverfiles/left4dead2/bin/server_srv.so
#3  0xf73ea357 in CGameEventManager::FireEventIntern(IGameEvent*, bool, bool) ()
   from /home/l4d2server/serverfiles/bin/engine_srv.so
#4  0xf73ea648 in CGameEventManager::FireEvent(IGameEvent*, bool) ()
   from /home/l4d2server/serverfiles/bin/engine_srv.so
#5  0xe93ad25c in __SourceHook_FHCls_IGameEventManager2FireEvent0::Func(IGameEvent*, bool) ()
   from /home/l4d2server/serverfiles/left4dead2/addons/sourcemod/bin/sourcemod.2.l4d2.so

I googled CSteamWorksGameStatsUploader and found a post from people using Sourcemod saying disabling Steamworks/stats on the server may fix the issue, so it might be related to an issue with Steam's stats

Looking at his debug.log it looks like it might be the same issue from 2016 still happening

Also #5 on my list references sourcemod, however I've already tested both on a VPS with just l4d2 installed (no mods installed, just the OS + L4D2 server) as well as my current server with the addons folder renamed (no mods loaded) and the crash occurs regardless

[rdroimfd]

The suggestion mentioned on the aforementioned forum thread solves the crashes, so this appears to definitely be related to the stats uploader (and for some reason only during new map loads)

sm_cvar steamworks_immediate_upload 0
sm_cvar steamworks_stats_disable 1

With these in server.cfg, no more segmentation faults, I've tested it about 20 times loading new maps. However it requires stat reporting to be disabled, so hopefully it is only temporary

[rdroimfd]

Thought this might be related to running the server with LinuxGSM; Installed on a new VM with steamcmd but I just started getting the same crashes after not getting them for a few minutes, so it must not be.

[iluno90]

I am having this crash on my server as well, didn't know it could be avoided by disabling stats until now. I added the sourcemod command to my settings which disables stats I think, and it stopped crashing.

Normally I just restart the server before playing to avoid it, but today for the first time the server was crashing and rebooting itself AFTER I'd done that. Happened twice in a row, with the same team each time, we had the same 3 people in the lobby, then as soon as we loaded into the server right after it said we'd connected, it crashed and rebooted.

But as soon as I made a solo lobby and started it that way, it loaded normally without crashing, and the 2 other people could join without causing the server crash. So it's weird the crashes only ever happen on new games started.

Here is what it shows if I crash with debug mode:

Thread 1 "srcds_linux" received signal SIGSEGV, Segmentation fault.
0xeded0a2e in CSteamWorksGameStatsUploader::GetInterface() ()
   from /home/surv/serverfiles/left4dead2/bin/server_srv.so
(gdb)

If I press 'l' command it says /valve/build/targets/src/gcc-4.6.1/libgcc/../gcc/libgcc2.c: No such file or directory.

And pressing c for continue:

Continuing.
[Thread 0xe4459b40 (LWP 4604) exited]
[Thread 0xe4358b40 (LWP 4600) exited]
[Thread 0xe4257b40 (LWP 4598) exited]
[Thread 0xe76c8b40 (LWP 4581) exited]
[Thread 0xe87b1b40 (LWP 4580) exited]
[Thread 0xe9651b40 (LWP 4578) exited]

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.

[iluno90]

Found a few more posts about this https://forums.alliedmods.net/showthread.php?t=294542 https://forum.myarena.ru/index.php?/topic/41209-khelp-server-krashitsia-kazhdyi-den/page-3 (in Russian)

I also did some more testing, ran server verification to see if maybe something is getting cached/changed causing the segfaults but it didn't fix anything.

I'm not sure if this is related or not, but in 'Steam/logs/stats_log.txt' there are a lot of lines like this:

[2018-12-03 15:57:23] [AppID 550] CAPIJobStoreUserStats::BInit() - no stats found, aborting

Those lines seem to happen whenever someone leaves the server (not during the segfaults themselves, just when they disconnect).

[NoisyFlake]

Thank you so much for documenting the two cvar fixes. I've been searching the internet for days and almost nobody seems to experience this issue. I've tested it on three different VPS (running Ubuntu 16.04, 18.04 and 20.04) and usually every second map load would result in a segmentation fault. This finally fixes it!

Valve should really take a look at this, since it's still happening in 2020, even after the Last Stand update.

[iluno90]

Sorry to see this is still happening to some people, I ended up making a script that automatically restarts my L4D2's game server whenever everyone has left the server and there are no people connected for a few seconds (and there is no one "reserved" or whatever as in the case when someone votes to change the map. Changing the level this way doesn't seem to trigger the crashes). That way stats still get reported and I don't get any crashing, the only downside is that if you leave the server and want to rejoin you have to wait ~5-10 more seconds while it's restarting. I did try disabling it to see if it's still happening after the recent Last Stand updates and it does still happen to me as well.