Server running out of edicts due to Viewmodels overhaul (?)

[awillinger]

Background: I am hosting a (fairly) popular classwars + randomizer server which locks all players on a team to the same class and gives them randomized weapons.

One feature of the classwars plugin is that it ForcePlayerSuicide's everyone on round start to properly reset everything. After a while, the server crashes with the infamous "ED_ALLOC: no free edicts" error.

When setting sv_lowedict_action to dump all edicts before crashing, the following is output (cut to the entities with most edicts):

L 05/07/2023 - 01:23:36: (4.54%) 93 spotlight_end
L 05/07/2023 - 01:23:36: (4.54%) 93 beam
L 05/07/2023 - 01:23:36: (8.35%) 171    tf_wearable
L 05/07/2023 - 01:23:36: (8.54%) 175    path_track
L 05/07/2023 - 01:23:36: (28.06%) 575   tf_wearable_vm

The path_track stuff is from the map (Upward) and to be expected. The last one, tf_wearable_vm, however, is not. That one appears to be fallout from #110. It seems like a lot of those entities are created but not properly cleaned up, at least in some cases, as the server (thankfully) only crashes 1-2 times/day while being full for ~12 hours. Maybe it's also some weird interaction caused by the mass suicide on round start.

I am also running another, far more popular classwars servers with the same plugins installed, minus randomizer, which crashes very very rarely and only because of stringtable limits.

Plugin info:

Filename: randomizer.smx
Title: Randomizer (Gamemode where everyone plays as random class with random weapons)
Author: 42
Version: 1.10.1.254
URL: https://github.com/FortyTwoFortyTwo/Randomizer
Status: running
Timestamp: 03/01/2023 22:21:53
Hash: 0c9bdebf991afff9cb82964ac87833b7

Configuration - basically default par the following settings:

randomizer_class ""
randomizer_cosmetics ""

Example crash link: https://crash.limetech.org/qgayjpusui7i

If you need any more information, feel free to ask!

[FortyTwoFortyTwo]

I could not figure out myself how one could create so many tf_wearable_vm to crash, which suggests me that it's classwars and/or ForcePlayerSuicide thats causing it, are either of these plugins public for me to try reproduce it?

Also im not sure why you would need a classwars plugin when randomizer convar allows you to do the same kind of thing.

This would make everyone for each teams to be same class. randomizer_class "trigger=@all group=@blue action=round same=1, trigger=@all group=@red action=round same=1"

And this would make everyone be same class regardless of team. randomizer_class "trigger=@all group=@all action=round same=1"

[awillinger]

For ForcePlayerSuicide I was referring to the SP function the classwars plugin calls when a class has been selected. The plugin itself is not public, but if needed, I can create a private repo and add you as a reader to it.

The classwars plugin also provides the !surrender function, which I didn't want to move into its own plugin back when I setup the server. Apart from the crashes it's working fine (par the infinite uber bug).

[FortyTwoFortyTwo]

I'm still having a hard time figuring out how any plugin would cause this cleanup failures, ViewModels_GetFromClient and ViewModels_DeleteFromClient already ensure it should never happen, unless if another plugin were to do something with tf_wearable_vm when created or spawned.

I see it happen during a new round when tf_wearable_vm gets deleted and regenerated at the same time in a full server, but 575 of tf_wearable_vm is way above my expectations that it would ever happen.

[awillinger]

Yea, I am really confused as well, especially, as the classwars plugin basically only forces players into a class and provides the surrender command and doesn't even remotely mess with tf_wearable_vm stuff.

Maybe the following problematic:

TF2_SetPlayerClass(client, GetTeamClass(team));

if(!forceApply)
{
return;
}

ForcePlayerSuicide(client);
TF2_RespawnPlayer(client);

forceApply is true on round start, and the crashes (as far as I have seen) only occur during that time. The ForcePlayerSuicide was a quick fix for an exploit, but imo isn't really needed anymore, at least not on round start. Will remove it and see if it resolves the issues...

[awillinger]

Seems like the removal of ForcePlayerSuicide didn't really change anything. The server was stable for 2 days at 28 slots, now, after increasing it to 32 players it started crashing again. Yesterday it even died after 4 hours (was mostly full during that time).

My only other guess is some weird side effects which occur when TF2_RespawnPlayer is called. The plugin itself does not mess with anything item-related otherwise.

On the Randomizer code side I couldn't really notice anything suspicious, apart from https://github.com/FortyTwoFortyTwo/Randomizer/blob/master/scripting/randomizer/viewmodels.sp#L29 where the RemoveEntity call will be skipped if for some reason the new weapon has the same modelIndex as the old... but then nothing gets created anyways. Kind of at a loss here.

My only other idea would be to kill all tf_wearable / tf_wearable_vm entities when a new class is selected and the player is about to be respawned (so right before TF2_RespawnPlayer). Not sure if it would cause issues on the Randomizer side, but as far as I can see there are numerous IsValidEntity calls, so it should be fine?

[FortyTwoFortyTwo]

I've made a new branch to try out a new method on this, it might catch any dupe viewmodels. See if this helps anything for you. https://github.com/FortyTwoFortyTwo/Randomizer/actions/runs/4973851881

[awillinger]

Alright, uploaded to the server, might have to restart the server tomorrow to clear out all "extra" entities. Will check back in a few days.

[awillinger]

Sadly, didn't seem to change much. While the server stayed online for 24 hours after I updated the plugin, it crashed twice again yesterday.

Crash dumps: https://crash.limetech.org/4lemdgzmotuw https://crash.limetech.org/cub7jqg3ulv5

[FortyTwoFortyTwo]

From all of the crashes you had, what was the highest and minimum edict count on tf_wearable_vm when it crashed?

[awillinger]

After I wrote my last message, the server was again online for more than 36 hours before crashing last night at ~4 AM.

From the last 7 crashes (only enabled edict spewing a few days ago), the highest was 638, the lowest 575. For tf_wearable itself the highest was 218, the lowest 184.

Direct output:

L 05/08/2023 - 21:00:56: (8.98%) 184    tf_wearable
L 05/08/2023 - 21:00:56: (28.06%) 575   tf_wearable_vm

L 05/13/2023 - 00:54:33: (10.64%) 218   tf_wearable
L 05/13/2023 - 00:54:33: (29.67%) 608   tf_wearable_vm

L 05/13/2023 - 20:42:07: (9.91%) 203    tf_wearable
L 05/13/2023 - 20:42:07: (29.04%) 595   tf_wearable_vm

L 05/14/2023 - 00:53:28: (10.05%) 206   tf_wearable
L 05/14/2023 - 00:53:28: (29.92%) 613   tf_wearable_vm

L 05/15/2023 - 23:18:23: (10.54%) 216   tf_wearable
L 05/15/2023 - 23:18:23: (30.01%) 615   tf_wearable_vm

L 05/16/2023 - 02:54:19: (10.25%) 210   tf_wearable
L 05/16/2023 - 02:54:19: (29.58%) 606   tf_wearable_vm

L 05/18/2023 - 02:10:42: (9.18%) 188    tf_wearable
L 05/18/2023 - 02:10:42: (31.14%) 638   tf_wearable_vm

[awillinger]

Some more crashes after the server was stable for 2 days: https://crash.limetech.org/x43drxgqomt5 https://crash.limetech.org/63upawcb3jcy

Interestingly, when the server crashes, it always appear to crash twice with only a few hours (2-6) between. After that, it stays online for several days...

Edict counts (in order): L 05/19/2023 - 22:48:06: (10.64%) 218 tf_wearable L 05/19/2023 - 22:48:06: (31.48%) 645 tf_wearable_vm

L 05/20/2023 - 00:47:36: (9.71%) 199 tf_wearable L 05/20/2023 - 00:47:36: (29.38%) 602 tf_wearable_vm

[awillinger]

Update: I have now written a plugin which detours Sys_Error_Internal and prints various statistics about tf_wearable_vm.

Relevant part (player part snipped as not relevant in this case):

int totalCount = 0, unsetCount = 0, invalidCount = 0;
int playerCount[MAXPLAYERS+1];

int entity = INVALID_ENT_REFERENCE;
while((entity = FindEntityByClassname(entity, "tf_wearable_vm")) != INVALID_ENT_REFERENCE)
{
    totalCount++;

    int owner = GetEntPropEnt(entity, Prop_Send, "m_hOwnerEntity");
    if(owner == -1)
    {
        unsetCount++;
        continue;
    }

    if(!IsValidEntity(owner))
    {
        invalidCount++;
        continue;
    }

    playerCount[owner]++;
}

LogMessage("Overall counts (total/unset/invalid): %d/%d/%d", totalCount, unsetCount, invalidCount);
PrintToServer("Overall counts (total/unset/invalid): %d/%d/%d", totalCount, unsetCount, invalidCount);

LogMessage("Per player counts: ");
PrintToServer("Per player counts: ");

Output from the latest crash:

20601 | 11635.154057 | Overall counts (total/unset/invalid): 622/562/0
20602 | 11635.154087 | Per player counts:

So it seems like the crash is caused by too many tf_wearable_vm instances existing without an owner set. Not sure how it got like that, my only guess is that they are not getting properly destroyed when their parent dies or when a player disconnects / changes class?

[FortyTwoFortyTwo]

I believe I found the problem, it's not related to randomizer's custom viewmodel, but it's weapons using extra wearables that's not cleared properly.

I had to buy a cheap botkiller weapon to test it out, this build should hopefully be fixed that you could test it out. https://github.com/FortyTwoFortyTwo/Randomizer/actions/runs/5046865254

[awillinger]

Ah, that would also explain why the server sometimes crashed again after 2 hours, while it stayed online for more than a day previously. If noone with such special weapons joined it stayed stable...

Installed the new version, will report back.

[awillinger]

So that new version caused the server to basically immediately crash after ~30 minutes, as it appears like the plugin never deletes the weapons, causing an edict overflow with that.

Excerpt from sv_dump_edicts:

28 tf_weapon_bat
2 tf_weapon_bat_fish
3 tf_weapon_bat_giftwrap
1 tf_weapon_bat_wood
18 tf_weapon_bonesaw
3 tf_weapon_bottle
3 tf_weapon_breakable_sign
18 tf_weapon_buff_item
2 tf_weapon_builder
4 tf_weapon_cannon
3 tf_weapon_charged_smg
2 tf_weapon_cleaver
15 tf_weapon_club
5 tf_weapon_compound_bow
3 tf_weapon_crossbow
5 tf_weapon_drg_pomson
36 tf_weapon_fireaxe
29 tf_weapon_fists
15 tf_weapon_flamethrower
14 tf_weapon_flaregun
4 tf_weapon_flaregun_revenge
7 tf_weapon_grenadelauncher
2 tf_weapon_handgun_scout_primary
15 tf_weapon_handgun_scout_secondary
51 tf_weapon_invis
8 tf_weapon_jar
3 tf_weapon_jar_gas
4 tf_weapon_jar_milk
2 tf_weapon_katana
17 tf_weapon_knife
6 tf_weapon_laser_pointer
22 tf_weapon_lunchbox
7 tf_weapon_lunchbox_drink
4 tf_weapon_mechanical_arm
25 tf_weapon_medigun
26 tf_weapon_minigun
6 tf_weapon_parachute_primary
5 tf_weapon_parachute_secondary
6 tf_weapon_particle_cannon
51 tf_weapon_pda_spy
7 tf_weapon_pep_brawler_blaster
17 tf_weapon_pipebomblauncher
4 tf_weapon_pistol_scout
2 tf_weapon_raygun
22 tf_weapon_revolver
7 tf_weapon_robot_arm
27 tf_weapon_rocketlauncher
11 tf_weapon_rocketlauncher_airstrike
6 tf_weapon_rocketlauncher_directhit
7 tf_weapon_rocketlauncher_fireball
6 tf_weapon_rocketpack
6 tf_weapon_sapper
13 tf_weapon_scattergun
4 tf_weapon_sentry_revenge
6 tf_weapon_shotgun_building_rescue
3 tf_weapon_shotgun_hwg
9 tf_weapon_shotgun_primary
5 tf_weapon_shotgun_pyro
9 tf_weapon_shotgun_soldier
27 tf_weapon_shovel
8 tf_weapon_slap
8 tf_weapon_smg
19 tf_weapon_sniperrifle
3 tf_weapon_sniperrifle_classic
8 tf_weapon_sniperrifle_decap
6 tf_weapon_soda_popper
1 tf_weapon_spellbook
5 tf_weapon_stickbomb
22 tf_weapon_sword
10 tf_weapon_syringegun_medic
15 tf_weapon_wrench
55 tf_wearable
1 tf_wearable_campaign_item
3 tf_wearable_demoshield
1 tf_wearable_razorback

Also, server chat:

Crash URL: https://crash.limetech.org/y6dkqffyl77c

The problem appears to still exist with that version too:

Overall counts (total/unset/invalid): 258/215/0

I wonder if we still need to call RemoveEntity / AcceptEntityInput("Kill") when removing weapons.

Rolled back to the previous version for now.

[FortyTwoFortyTwo]

Forgor something obvious to add 💀. Very obvious reason why i would commit to a new branch, anyway this new build should do. https://github.com/FortyTwoFortyTwo/Randomizer/actions/runs/5049953776

[awillinger]

Seems like even this fix didn't change much, still crashed 3 times last night. And still with 570+ invalid wearables.

On 22 May 2023, 22:09, at 22:09, 42 @.***> wrote:

Forgor something obvious to add 💀. Very obvious reason why i would commit to a new branch, anyway this new build should do. https://github.com/FortyTwoFortyTwo/Randomizer/actions/runs/5049953776

-- Reply to this email directly or view it on GitHub: https://github.com/FortyTwoFortyTwo/Randomizer/issues/115#issuecomment-1557897753 You are receiving this because you authored the thread.

Message ID: @.***>

[FortyTwoFortyTwo]

Egh, don't really have a clue now on what else could be causing it. Could you look into seeing what world model the wearables and vm have it set? Might give clues on whenever if a specific model has appeared a lot from it.

[awillinger]

Would that be the netprop "m_nModelIndex"?

[FortyTwoFortyTwo]

Yeah, this stock is simple enough to get the model

stock void GetEntityModel(int iEntity, char[] sModel, int iMaxSize, const char[] sProp = "m_nModelIndex")
{
    int iIndex = GetEntProp(iEntity, Prop_Send, sProp);
    int iTable = FindStringTable("modelprecache");
    ReadStringTable(iTable, iIndex, sModel, iMaxSize);
}

[awillinger]

So, after 4 days of continuous uptime, the server finally managed to crash again.

Here's the output with counts per model:

455151 | 407419.729376 | Unset models with counts:
-- | -- | --
455152 | 407419.729398 | - : 5
455153 | 407419.729415 | - models/workshop/weapons/c_models/c_reserve_shooter/c_reserve_shooter.mdl: 4
455154 | 407419.729429 | - models/workshop/weapons/c_models/c_paintrain/c_paintrain.mdl: 2
455155 | 407419.729443 | - models/workshop_partner/weapons/c_models/c_sd_cleaver/c_sd_cleaver.mdl: 4
455156 | 407419.729457 | - models/workshop/weapons/c_models/c_pro_smg/c_pro_smg.mdl: 1
455157 | 407419.729472 | - models/weapons/c_models/c_medigun/c_medigun.mdl: 4
455158 | 407419.729487 | - models/workshop/weapons/c_models/c_scatterdrum/c_scatterdrum.mdl: 3
455159 | 407419.729501 | - models/weapons/c_models/urinejar.mdl: 2
455160 | 407419.729517 | - models/workshop/weapons/c_models/c_gatling_gun/c_gatling_gun.mdl: 2
455161 | 407419.729543 | - models/weapons/c_models/c_ambassador/c_ambassador.mdl: 5
455162 | 407419.729573 | - models/weapons/c_models/c_tfc_sniperrifle/c_tfc_sniperrifle.mdl: 6
455163 | 407419.729598 | - models/weapons/c_models/c_fireaxe_pyro/c_fireaxe_pyro.mdl: 2
455164 | 407419.729621 | - models/weapons/c_models/c_hippocrates_bust/c_hippocrates_bust.mdl: 1
455165 | 407419.729643 | - models/workshop/weapons/c_models/c_winger_pistol/c_winger_pistol.mdl: 7
455166 | 407419.729665 | - models/weapons/c_models/c_energy_drink/c_energy_drink.mdl: 4
455167 | 407419.729685 | - models/weapons/c_models/c_rocketpack/c_rocketpack.mdl: 1
455168 | 407419.729713 | - models/workshop/weapons/c_models/c_letranger/c_letranger.mdl: 1
455169 | 407419.729751 | - models/weapons/c_models/c_demo_arms.mdl: 12
455170 | 407419.729776 | - models/workshop/weapons/c_models/c_amputator/c_amputator.mdl: 6
455171 | 407419.729800 | - models/workshop/weapons/c_models/c_jag/c_jag.mdl: 1
455172 | 407419.729829 | - models/workshop/weapons/c_models/c_holymackerel/c_holymackerel.mdl: 2
455173 | 407419.729855 | - models/workshop/weapons/c_models/c_russian_riot/c_russian_riot.mdl: 16
455174 | 407419.729878 | - models/workshop/weapons/c_models/c_sledgehammer/c_sledgehammer.mdl: 5
455175 | 407419.729903 | - models/workshop/weapons/c_models/c_trenchgun/c_trenchgun.mdl: 13
455176 | 407419.729926 | - models/workshop/weapons/c_models/c_liberty_launcher/c_liberty_launcher.mdl: 10
455177 | 407419.729948 | - models/workshop/weapons/c_models/c_pep_scattergun/c_pep_scattergun.mdl: 2
455178 | 407419.729972 | - models/workshop_partner/weapons/c_models/c_dex_shotgun/c_dex_shotgun.mdl: 4
455179 | 407419.729994 | - models/weapons/c_models/c_bugle/c_bugle.mdl: 5
455180 | 407419.730021 | - models/workshop_partner/weapons/c_models/c_dex_arm/c_dex_arm.mdl: 4
455181 | 407419.730046 | - models/workshop/weapons/c_models/c_drg_thirddegree/c_drg_thirddegree.mdl: 3
455182 | 407419.730073 | - models/weapons/c_models/c_scout_arms.mdl: 6
455183 | 407419.730097 | - models/weapons/c_models/c_minigun/c_minigun_natascha.mdl: 2
455184 | 407419.730143 | - models/workshop/weapons/c_models/c_rift_fire_axe/c_rift_fire_axe.mdl: 3
455185 | 407419.730173 | - models/weapons/c_models/c_scottish_resistance/c_scottish_resistance.mdl: 2
455186 | 407419.730199 | - models/weapons/c_models/c_boxing_gloves/c_boxing_gloves.mdl: 2
455187 | 407419.730221 | - models/weapons/c_models/c_proto_syringegun/c_proto_syringegun.mdl: 20
455188 | 407419.730246 | - models/workshop/weapons/c_models/c_lochnload/c_lochnload.mdl: 18
455189 | 407419.730270 | - models/weapons/c_models/c_stickybomb_launcher/c_stickybomb_launcher.mdl: 6
455190 | 407419.730291 | - models/weapons/c_models/c_bottle/c_bottle.mdl: 2
455191 | 407419.730314 | - models/weapons/c_models/c_machete/c_machete.mdl: 2
455192 | 407419.730337 | - models/workshop/weapons/c_models/c_shortstop/c_shortstop.mdl: 5
455193 | 407419.730358 | - models/weapons/c_models/c_slapping_glove/w_slapping_glove.mdl: 2
455194 | 407419.730381 | - models/workshop/weapons/c_models/c_drg_cowmangler/c_drg_cowmangler.mdl: 2
455195 | 407419.730405 | - models/workshop/weapons/c_models/c_drg_phlogistinator/c_drg_phlogistinator.mdl: 4
455196 | 407419.730426 | - models/workshop/weapons/c_models/c_claidheamohmor/c_claidheamohmor.mdl: 2
455197 | 407419.730448 | - models/weapons/c_models/c_soldier_arms.mdl: 57
455198 | 407419.730471 | - models/workshop/weapons/c_models/c_xms_giftwrap/c_xms_giftwrap.mdl: 3
455199 | 407419.730494 | - models/workshop/weapons/c_models/c_kingmaker_sticky/c_kingmaker_sticky.mdl: 1
455200 | 407419.730522 | - models/weapons/c_models/c_heavy_arms.mdl: 17
455201 | 407419.730548 | - models/weapons/c_models/c_knife/c_knife.mdl: 3
455202 | 407419.730576 | - models/workshop/weapons/c_models/c_xms_gloves/c_xms_gloves.mdl: 2
455203 | 407419.730599 | - models/workshop/weapons/c_models/c_back_scratcher/c_back_scratcher.mdl: 2
455204 | 407419.730621 | - models/workshop/weapons/c_models/c_fists_of_steel/c_fists_of_steel.mdl: 1
455205 | 407419.730644 | - models/workshop/weapons/c_models/c_pep_pistol/c_pep_pistol.mdl: 1
455206 | 407419.730666 | - models/weapons/c_models/c_engineer_arms.mdl: 46
455207 | 407419.730688 | - models/weapons/c_models/c_pyro_arms.mdl: 5
455208 | 407419.730715 | - models/weapons/c_models/c_sniper_arms.mdl: 3
455209 | 407419.730737 | - models/weapons/c_models/c_flameball/c_flameball.mdl: 1
455210 | 407419.730759 | - models/weapons/c_models/c_smg/c_smg.mdl: 9
455211 | 407419.730781 | - models/workshop/weapons/c_models/c_buffalo_steak/c_buffalo_steak.mdl: 2
455212 | 407419.730804 | - models/workshop_partner/weapons/c_models/c_shogun_warhorn/c_shogun_warhorn.mdl: 1
455213 | 407419.730827 | - models/workshop/weapons/c_models/c_powerjack/c_powerjack.mdl: 1
455214 | 407419.730850 | - models/weapons/c_models/c_proto_medigun/c_proto_medigun.mdl: 1
455215 | 407419.730880 | - models/workshop/weapons/c_models/c_battleaxe/c_battleaxe.mdl: 1
455216 | 407419.730909 | - models/workshop/weapons/c_models/c_demo_sultan_sword/c_demo_sultan_sword.mdl: 8
455217 | 407419.730934 | - models/workshop/weapons/c_models/c_xms_cold_shoulder/c_xms_cold_shoulder.mdl: 2
455218 | 407419.730957 | - models/weapons/c_models/c_sniperrifle/c_sniperrifle.mdl: 3
455219 | 407419.730978 | - models/weapons/c_models/c_revolver/c_revolver.mdl: 6
455220 | 407419.731001 | - models/weapons/c_models/c_sapper/c_sapper.mdl: 2
455221 | 407419.731027 | - models/weapons/c_models/c_pistol/c_pistol.mdl: 1
455222 | 407419.731048 | - models/weapons/c_models/c_csgo_awp/c_csgo_awp.mdl: 1
455223 | 407419.731071 | - models/weapons/c_models/c_shovel/c_shovel.mdl: 1
455224 | 407419.731093 | - models/weapons/c_models/c_ubersaw/c_ubersaw.mdl: 3
455225 | 407419.731116 | - models/weapons/c_models/c_medic_arms.mdl: 4
455226 | 407419.731138 | - models/weapons/c_models/c_bow/c_bow.mdl: 6
455227 | 407419.731159 | - models/workshop/weapons/c_models/c_quadball/c_quadball.mdl: 14
455228 | 407419.731183 | - models/workshop/weapons/c_models/c_drg_wrenchmotron/c_drg_wrenchmotron.mdl: 5
455229 | 407419.731209 | - models/workshop/weapons/c_models/c_acr_hookblade/c_acr_hookblade.mdl: 1
455230 | 407419.731239 | - models/workshop/weapons/c_models/c_spikewrench/c_spikewrench.mdl: 1
455231 | 407419.731264 | - models/workshop/weapons/c_models/c_drg_manmelter/c_drg_manmelter.mdl: 10
455232 | 407419.731290 | - models/weapons/c_models/c_minigun/c_minigun.mdl: 1
455233 | 407419.731313 | - models/workshop/weapons/c_models/c_boston_basher/c_boston_basher.mdl: 3
455234 | 407419.731338 | - models/workshop/weapons/c_models/c_market_gardener/c_market_gardener.mdl: 4
455235 | 407419.731366 | - models/weapons/c_models/c_pickaxe/c_pickaxe.mdl: 1
455236 | 407419.731393 | - models/workshop/weapons/c_models/c_bazaar_sniper/c_bazaar_sniper.mdl: 2
455237 | 407419.731419 | - models/weapons/c_models/c_axtinguisher/c_axtinguisher_pyro.mdl: 5
455238 | 407419.731446 | - models/weapons/c_models/c_spy_arms.mdl: 17
455239 | 407419.731473 | - models/weapons/c_models/c_sticky_jumper/c_sticky_jumper.mdl: 5
455240 | 407419.731499 | - models/workshop_partner/weapons/c_models/c_canton/c_canton.mdl: 1
455241 | 407419.731526 | - models/workshop/weapons/c_models/c_pro_rifle/c_pro_rifle.mdl: 1
455242 | 407419.731553 | - models/workshop/weapons/c_models/c_detonator/c_detonator.mdl: 3
455243 | 407419.731580 | - models/workshop_partner/weapons/c_models/c_sd_neonsign/c_sd_neonsign.mdl: 2
455244 | 407419.731606 | - models/weapons/c_models/c_wooden_bat/c_wooden_bat.mdl: 2
455245 | 407419.731638 | - models/workshop_partner/weapons/c_models/c_shogun_katana/c_shogun_katana_soldier.mdl: 4
455246 | 407419.731664 | - models/weapons/c_models/c_claymore/c_claymore.mdl: 1
455247 | 407419.731698 | - models/weapons/c_models/c_shotgun/c_shotgun.mdl: 11
455248 | 407419.731731 | - models/workshop_partner/weapons/c_models/c_sd_sapper/c_sd_sapper.mdl: 1
455249 | 407419.731758 | - models/weapons/c_models/c_flamethrower/c_backburner.mdl: 4
455250 | 407419.731783 | - models/workshop/weapons/c_models/c_tele_shotgun/c_tele_shotgun.mdl: 5
455251 | 407419.731810 | - models/workshop/weapons/c_models/c_rift_fire_mace/c_rift_fire_mace.mdl: 5
455252 | 407419.731838 | - models/workshop/weapons/c_models/c_sydney_sleeper/c_sydney_sleeper.mdl: 1
455253 | 407419.731865 | - models/workshop/weapons/c_models/c_eternal_reward/c_eternal_reward.mdl: 5
455254 | 407419.731893 | - models/workshop/weapons/c_models/c_madmilk/c_madmilk.mdl: 2
455255 | 407419.731919 | - models/workshop/weapons/c_models/c_uberneedle/c_uberneedle.mdl: 2
455256 | 407419.731947 | - models/workshop_partner/weapons/c_models/c_dex_sniperrifle/c_dex_sniperrifle.mdl: 2
455257 | 407419.731973 | - models/workshop/weapons/c_models/c_scorch_shot/c_scorch_shot.mdl: 1
455258 | 407419.731999 | - models/workshop/weapons/c_models/c_bear_claw/c_bear_claw.mdl: 1
455259 | 407419.732026 | - models/workshop/weapons/c_models/c_snub_nose/c_snub_nose.mdl: 6
455260 | 407419.732051 | - models/weapons/c_models/c_frontierjustice/c_frontierjustice.mdl: 1
455261 | 407419.732077 | - models/workshop/weapons/c_models/c_atom_launcher/c_atom_launcher.mdl: 10

Doesn't seem like one singular item is present a lot of times, but a bunch:

• Soldier Arms: 57x (!)
• Engineer Arms: 46x
• Overdose: 20x
• Loch'n'Load
• Family Business: 16x

Just to name a few.

[awillinger]

Some more crashes from 19 and 1 hours ago (at the time of posting): https://pastebin.com/C4hKHdav https://pastebin.com/1v4Sjv7g

Decided to put those in pastebins so that the page still remains scrollable.

[FortyTwoFortyTwo]

While I was investigating on how leaked tf_wearable_vm could happen, I accidently found myself an exploit on leaked tf_weapon and tf_wearable leaked, with the fix now up in master. I have no idea if you had encountered this issue, but give it a try to see if it would fix tf_wearable_vm, or at the very least the amount of tf_wearable made.

Otherwise regarding the tf_wearable_vm models, I can see the pattern that ~30% of unowned viewmodels are arms, and that 2 of the class arms count are a lot more bigger compared to other classes, which I assume is because of your class plugin. Otherwise I couldn't find myself any reasonable conclusion from this model debug.

[awillinger]

Hmm, but the version in master is missing the two other fixes currently. Is there anything speaking against merging the feature branch into it?

[FortyTwoFortyTwo]

Both "fixes" don't seems to help anything from solving this(?), so there is no need to have it in master.

[awillinger]

True that, will try with the version from master now.

[awillinger]

To give an update to this: I have now modified the classwars plugin so that the assignment of classes on round start happens in the following frame, meaning that the call to TF2_RespawnPlayer only ever happens there as well.

Basically: teamplay_round_start -> RequestFrame -> TF2_SetPlayerClass && TF2_RespawnPlayer

Additionally, I have implemented a plugin which "attaches" itself to each created tf_wearable_vm and checks if m_hOwnerEntity == -1 every second. If that condition is true, the entity is deleted.

So far, the server hasn't crashed because of no free edicts for more than 4 days, while being constantly full for the past 2:

As you may, however, notice, the server is still crashing occasionally, but this time because of "CTFWeaponBase::ReloadSinglyPostFrame()", which always occurs when trying to reload some kind of pipe launcher (CTFPipebombLauncher::ItemPostFrame()).

Recent crashes: https://crash.limetech.org/tnvcudleycpa https://crash.limetech.org/6pbiq7qzh7pj https://crash.limetech.org/zwrrvnnhjkyr

Judging from https://github.com/VSES/SourceEngine2007/blob/master/src_main/game/shared/tf/tf_weaponbase.cpp#L934 it appears as though the "owner" somehow becomes null, causing a null pointer (eax is 0):

Not sure how to start debugging this. My first suspect would be someone reloading on round end right before the new one starts.

[FortyTwoFortyTwo]

For the pipe launcher crash, it seems to be that client has an active weapon (m_hActiveWeapon) equipped, but does not have owner (m_hOwnerEntity) set, as ItemPostFrame is called from CBasePlayer::ItemPostFrame. Are you able to track down on when and why m_hOwnerEntity has been set to null, and whenever if weapon's owner as null is related to the whole tf_wearable_vm issue?

[awillinger]

Well, the only hint I had was after I also added tf_wearable_vm logging on round end. Right before the crash, the logging was triggered twice - once with 0 invalid entities, then maybe 1-2 seconds later with 500+. This was also the reason why I suspected a timing issue and added the RequestFrame stuff.

I also thought about adding SendProxy and hooking m_hOwnerEntity changes, but it appears like none of the pre-compiled binaries work anymore or not without crashing. Also, would still need some kind of way to get the caller of it, so basically Java's StackWalker.

[awillinger]

To finally give closure to this ticket: After changing the classwars plugin to change the class delayed by one frame, the server no longer crashed due to running out of edicts. The ReloadSinglyPostFrame errors are handled by a separate plugin which simply kills all weapons without any m_hOwnerEntity, those occur very rarely tho.

So, nothing to fix in this plugin after all.