Heap/stack corruption on map change following bot spawning

[geominorai]

Linux TF2 server segfaults on map change after spawning a puppet bot

TF2 srcds v4294355 Sourcemod v1.8.0.6040 Metamod 1.10.7-git961

Reproduction steps:

1. Start/restart server normally with a startup map (no need for players to even join the server)
2. Enter in server console: sv_cheats 1; bot -name SomeBOT -class soldier; sv_cheats 0
3. Enter in server console: changelevel (some other map)
4. Crashes showing in console log: corrupted size vs. prev_size

Repeatability:

• Almost always (> 70% of the time via reproduction steps on my server)
• Crash does not happen without spawning the bot
• Crash does not happen without Sourcemod loaded
• Crash does not happen with only Metamod loaded

Time frame:

• Noticed this on a plugin of mine that spawns bots after the Jungle update. Debugged for weeks to find out the same crash can happen without running any plugins at all.

Crash dump: https://crash.limetech.org/a4w7wssvn6z6 Additional log file: debug.log

Plugins loaded:

• [None] (tested with empty plugins folder)

Extensions loaded:

• bintools.ext.so
• game.tf2.ext.2.tf2.so
• sdkhooks.ext.2.tf2.so
• sdktools.ext.2.tf2.so
• updater.ext.so
• webternet.ext.so

[Fyren]

I can reproduce this on my Linux server. Beyond what was already said, the crash occurs on the latest 1.9 build and without any SM extensions loaded. Choice of map seems to matter in some way. I started the server on itemtest and running sv_cheats 1; bot -name SomeBOT -class soldier; sv_cheats 0; wait 60; changelevel pl_goldrush would always crash for me.

[geominorai]

If I manually kick the bot prior to switching the map, the crash does not seem to happen. However, it is too late by the time OnMapEnd is called because bots are automatically removed by the server at that point and the crash will happen at the start of the next map.

This also means I cannot predict when a map will end beforehand to use this workaround, and any plugin that calls SM's ForceChangeLevel function will crash the server on top of the normal map cycling.

[xNanochip]

Getting this issue with tf bots as well.

[Glubbable]

I can confirm that I am having this crash issue with regular bots as well. However, important thing to note for me is that this crash does not occure on a server running under Debian, but does occure if the server is running Ubuntu.

[Kenzzer]

Made a crash "avoider", https://forums.alliedmods.net/showthread.php?p=2585800 Won't fix the issue, but at least it should help people suffering from this. I would have loved to help fixing that issue but I've no idea where to look or start or what may even be the root problem, so I hope someone with will give us a proper fix a day.

[geominorai]

@Glubbable: Can confirm; it crashes on Ubuntu but also on Gentoo, which is what NFOservers managed Linux game servers also run.

[KyleSanderson]

Does this still happen after todays fix?

[Fyren]

There's practically zero chance this is related to the convar change. That only started after our change in early May.

[sigsegv-mvm]

To everyone following this thread: I highly recommend that you take a look at #1007 #1107 (and the fix in #1008 #1108). I'm pretty sure that I found the root cause of this problem, once and for all. Plus a fix for it.

Try a SourceMod rebuild with the latest commits and see if your heap corruption problems go away!

[sigsegv-mvm]

Oh... and oops, in my previous post I referenced the wrong issue/PR numbers. :roll_eyes:

Fixed now.

[asherkin]

Gonna go ahead and hope this was fixed by the PlayerManager changes 🤞