No free edicts crash

[x3m77rus]

After update on latest commit server began to crash every 20-30 mins with Engine error: ED_Alloc: no free edicts. I'm know this is pretty common issue on moded servers with big detailed maps and many precached models, but seems problem in weapons. For some reason too many edicts taken by tf_weapon and tf_viewmodel. Looks like not freed item handle in repeating timer, but SM not reporting about lack of handles. Reverted back on 0.3.0 and crashes stopped.

Recent server logs: l0314001.txt l0314002.txt l0314003.txt l0314004.txt l0314005.txt l0314006.txt

[x3m77rus]

Upd: Nope, crashes don't stop

[Kenzzer]

If crashes don't stop even if you moved to an old versions that means it's an other plugin, I would recommand you to install Accelerator if you have a linux server, and show me the crash dump. Actually we are not experiencing those crashes on Glub's servers.

[x3m77rus]

Here is crash dump https://crash.limetech.org/jfaiq2wpdvbt

[Kenzzer]

Hmm, from what I recall and the crash dump, it looks like the proxy's code is executed before it crash.

10 server_srv.so!CreateEntityByName(char const, int) + 0x3e 11 server_srv.so!CItemGeneration::SpawnItem(CEconItemView const, Vector const&, QAngle const&, char const) + 0x4b 12 server_srv.so!CTFPlayer::GiveNamedItem(char const, int, CEconItemView const, bool) + 0x90 13 tf2items.ext.2.ep2v.so!__SourceHook_MFHCls_MHook_GiveNamedItem::Func(char const, int, CEconItemView, bool) [extension.cpp:47 + 0x28] 14 server_srv.so!CTFPlayer::ManageRegularWeapons(TFPlayerClassData_t) + 0xe2f 15 server_srv.so!CTFPlayer::GiveDefaultItems() + 0x38 16 server_srv.so!CTFPlayer::InitClass() + 0x9f 17 server_srv.so!CTFPlayer::Spawn() + 0x356 18 server_srv.so!CTFPlayer::ForceRespawn() + 0x433 49 server_srv.so!CBasePlayer::ChangeTeam(int, bool, bool) + 0x14b 50 server_srv.so!CTFPlayer::ChangeTeam(int, bool, bool) + 0x14e 51 server_srv.so!CTFPlayer::ChangeTeam(int) + 0x2b

Try to update your tf2items.

And tell me if "proxies_zombies" is on 1 for one of your bosses.

I think EquipWearable vtable offests are outdated, I will disamble the server library and take a look on.

[x3m77rus]

Today server worked and still working without crashes for 1300 mins but with average 17-18 players. Yesterday crashed even with 10 players. I'm disabled plugin that giving some players attributes on weapons, such as killstreaks or no fall damage through tf2attributes. May this confront with proxies code? I'm think it can be a reason of crashes, but before tf2 update everything was fine for a long times without any crashes and even reboot. But proxies are disabled on my server. Weird.

Anyway thanks for explanation, now I'm know that it's not a mod issue. Also will try to update tf2items.

[x3m77rus]

Was updated tf2items and also changed two cvars: sv_lowedic_action 4 for changing to next map in mapcycle and sv_lowedict_threshold 128. Crashes begin being less frequent but still exist. https://crash.limetech.org/dytkr3j72gkz

You're said you're not experiencing those crashes, are you using plugins like https://forums.alliedmods.net/showthread.php?t=265902 or https://forums.alliedmods.net/showthread.php?t=186830 ?

[Kenzzer]

Yes we are using my plugin SEL on Glub's servers. But it makes a log when it delets something, and nothing about your entities. Oh and can you tell if "proxies_zombies" is on 1 for one of your bosses, it's really important!

[x3m77rus]

I also tried to use SEL, but then disabled, plugin reported, that entity limit is approaching, then begin to kill some entities and kills worldspawn, and then server crash. Yes, found "proxies zombies" set to 1. But "proxies" on this boss is set to 0. Just in case setted "proxies zombies" to 0.

[Kenzzer]

weird it tries to kill the world? I have probably not updated SEL on AM. I will do it later, and I looked at the vtable offsets for EquipWearable it's not outdated, so you should be able to use proxies_zombies. Can you post me a list of your plugins? ("sm plugins list" in server console)

[x3m77rus]

All dependencies, offsets, mm and sm versions always up to date, all servers runs with common gamedata and extensions folders, so if any of them will be outdated, any server, that dependence on them will crash or not work too. For example, FF2 is using EquipWearable too, in his own file of cource, but offsets the same, and works well.

Of cource, no problem Plugin list: 01 "SourceBans++" (1.5.4) by SourceBans Development Team, Sarabveer(VEER™) 02 "SourceSleuth" ((SB++) 1.5.4) by ecca, Sarabveer(VEER™) 03 "SourceBans Checker" ((SB++) 1.5.4) by psychonic, Ca$h Munny, Sarabveer(VEER™) 04 "SourceBans++: Admin Config Loader" (1.5.4) by AlliedModders LLC, Sarabveer(VEER™) 05 "Nextmap" (1.7.3-dev+5295) by AlliedModders LLC 06 "Player Commands" (1.7.3-dev+5295) by AlliedModders LLC 07 "Basic Commands" (1.7.3-dev+5295) by AlliedModders LLC 08 "Admin Help" (1.7.3-dev+5295) by AlliedModders LLC 09 "Basic Chat" (1.7.3-dev+5295) by AlliedModders LLC 10 "Reserved Slots" (1.7.3-dev+5295) by AlliedModders LLC 11 "Client Preferences" (1.7.3-dev+5295) by AlliedModders LLC 12 "Rock The Vote" (1.7.3-dev+5295) by AlliedModders LLC 13 "Map Nominations" (1.7.3-dev+5295) by AlliedModders LLC 14 "Fun Commands" (1.7.3-dev+5295) by AlliedModders LLC 15 "Basic Info Triggers" (1.7.3-dev+5295) by AlliedModders LLC 16 "Basic Votes" (1.7.3-dev+5295) by AlliedModders LLC 17 "Anti-Flood" (1.7.3-dev+5295) by AlliedModders LLC 18 "Fun Votes" (1.7.3-dev+5295) by AlliedModders LLC 19 "Basic Comm Control" (1.7.3-dev+5295) by AlliedModders LLC 20 "Admin Menu" (1.7.3-dev+5295) by AlliedModders LLC 21 "Sound Commands" (1.7.3-dev+5295) by AlliedModders LLC 22 "ExtendedComm" (3.0.8b) by Twisted|Panda 23 "[Any] Entity Logger" (1.0) by Tak (Chaosxk) 24 "X3M-Slender" (1.0) by X3M.77[RUS] 25 "Server Redirect: Ask connect with connect box" (0.1.0) by H3bus 26 "Slender Fortress" (0.3.1) by KitRifty, Benoist3012 27 "SourceMod Anti-Cheat" (0.8.6.0) by SMAC Development Team 28 "SMAC Rcon Locker" (0.8.6.0) by SMAC Development Team 29 "SMAC Eye Angle Test" (0.8.6.0) by SMAC Development Team 30 "SMAC Client Protection" (0.8.6.0) by SMAC Development Team 31 "SMAC Aimbot Detector" (0.8.6.0) by SMAC Development Team 32 "SMAC AutoTrigger Detector" (0.8.6.0) by SMAC Development Team 33 "SMAC Command Monitor" (0.8.6.0) by SMAC Development Team 34 "Trade Chat" (1.5.1) by Luki 35 "Cronjobs" (1.2) by Popoklopsi 36 "Server Redirect" (0.1.2) by H3bus 37 "MOTDgd Adverts" (2.3.5) by Blackglade and Ixel 38 "Stamm Feature RegenerateHP" (1.3.2) by Popoklopsi 39 "Stamm Feature Joinsound" (1.4.1) by Popoklopsi 40 "Stamm Feature Chat Colors" (1.0.1) by Popoklopsi 41 "Stamm Feature Killstreak" (2.0.0) by X3M.77[RUS] 42 "Stamm Feature MoreSpeed" (1.3.1) by Popoklopsi 43 "Stamm Feature SpawnHP" (1.4.1) by Popoklopsi 44 "Stamm Feature PlayerTrail" (1.3.1) by Popoklopsi 45 "Stamm Feature No Fall Damage" (2.0.0) by X3M.77[RUS] 46 "Stamm Feature No Reload" (1.3.1) by Popoklopsi 47 "Stamm" (2.2.8) by Popoklopsi 48 "NativeVotes Basic Commands" (1.0) by Powerlord and AlliedModders LLC 49 "Advertisements" (0.6) by Tsunami 50 "Spray Tracer No Menu" (v5.8b) by Nican132, CptMoore, Lebson506th 51 "Killstreak" (1.0) by Dr_Knuckles 52 "Block Trading" (1.2) by Matheus28 53 "NativeVotes" (0.8.3) by Powerlord 54 "Automatic Steam Update (SteamWorks)" (1.0) by Dr. McKay, Sarabveer(VEER™) 55 "TF2 Build Restrictions" (0.5.5) by Tsunami 56 "[Source 2009] Custom Chat Colors" (3.0.1) by Dr. McKay 57 "[TF2] Thirdperson" (2.1.0) by DarthNinja 58 "Simple Chat Processor (Redux)" (1.1.5) by Simple Plugins, Mini 59 "[TF2] TF2Attributes" (1.1.1) by FlaminSarge 60 "Players Votes" (1.5.0) by The Resident, pZv! 61 "Force HTML MOTDs" (1.0.5) by Twisted|Panda 62 "[TF2] Taunt Menu" (1.07) by FlaminSarge, Nighty, xCoderx, Crow 63 "NativeVotes MapChooser" (1.5.4) by AlliedModders LLC and Powerlord 64 "SourcePawn Navigation Mesh Parser" (1.0.3) by KitRifty 65 "X3M-Stats" (3.0) by X3M.77[RUS]

Extensions list: [01] Automatic Updater (1.7.3-dev+5295): Updates SourceMod gamedata files [02] Webternet (1.7.3-dev+5295): Extension for interacting with URLs [03] Equinox Console Cleaner (): Bite me alien boi [04] Accelerator (2.2.1): SRCDS Crash Handler [05] TF2Items (1.6.4): TF2 Item Modifier [06] TF2 Tools (1.7.3-dev+5295): TF2 extended functionality [07] BinTools (1.7.3-dev+5295): Low-level C/C++ Calling API [08] SDK Hooks (1.7.3-dev+5295): Source SDK Hooks [09] SDK Tools (1.7.3-dev+5295): Source SDK Tools [10] Top Menus (1.7.3-dev+5295): Creates sorted nested menus [11] Client Preferences (1.7.3-dev+5295): Saves client preference settings [12] SQLite (1.7.3-dev+5295): SQLite Driver [13] Regex (1.7.3-dev+5295): Provides regex natives for plugins [14] SteamTools (0.9.1+7202836): SteamWorks for SourceMod. [15] SteamWorks Extension (1.2): Exposes SteamWorks functions to Developers [16] DHooks (2.0.3): Dynamic Hooks [17] System2 (2.32): System Commands and Information [18] Socket (3.0.1): Socket extension for SourceMod [19] GeoIP (1.7.3-dev+5295): Geographical IP information [20] MySQL-DBI (1.7.3-dev+5295): MySQL driver implementation for DBI

Also for test, removed code, where clients are forced to respawn to call spawn event like here https://github.com/Benoist3012/Slender-Fortress-New-Versions/commit/9201fc51866440e5c05b8338a176d268aebc57b9

[Kenzzer]

Everything seems fine, perhaps "24 "X3M-Slender" (1.0) by X3M.77[RUS]", but it's your plugin, so I think the crash isn't from it. Have you turned on the lags compensation for players in pvp area?

[x3m77rus]

That plugin only makes a dynamic hostname for server such as hostname | Boss: bossname and prevents clients from spamming in chat lenny faces and etc. Almost all cvars are their defaults, so sf2_player_fakelagcompensation also are 0 by default.

[Kenzzer]

Then it's not SF2, a plugin is messing with player view models on your server. And I can't see which plugin is causing that on your plugins list. Perhaps taunt menu plugin?

[x3m77rus]

Okay, will try to setup a clean server with SF2 only for test. And what about of tfweapon* edicts? Is it normal, that so many edicts taken by wepons with same entity classes?

In my mind, for example, 32 players each in average with 3 weapons, yes spies and engies have more, but red team have melee only, so we may take weapons as 3 per player. One weapon - one viewmodel. 32 x 3 + 32 x 3 = 192 May even round them to 250 edicts to all weapons on server, but not to one class.

My reasoning may be foolish, but anyway.

[Kenzzer]

No you are right, it's around 192 weapon entities with a full server. But your problem isn't from the weapons, but from the view model entity, 500 view model entities on your server before crash. You should be around 30. I recommand you to load one by one your plugins on your clean server until you get a crash.

[x3m77rus]

Something is really weird. Clean server with sourcemod and SF2 only still crashing at least one time in a hour. But with different reasons of no free edicts. Last two times was 565 env_sprite, before 700+ tf_wearable. Looks like someone intentionally crashing server by some kind of exploit or bug, but i dont know how.

Also, is it matters that i compile plugin by myself?

[Kenzzer]

No you are absolutly free to do it. I think it's a problem with your tf2 items.

[x3m77rus]

TF2Items is 1.6.4-279 the latest build and other servers running with same file is working fine. So I will continue investigate this. Thanks for the tips and answers!

[Kenzzer]

Alright I will leave this issue "pending" until you find something. Feel free to close this issue whenever you want.

[x3m77rus]

Seems found a reason. Force HTML MOTDs (1.0.5) by Twisted|Panda is causing this. This plugin is blocking team change from spectators if client doesn't have html MOTD enabled. I'm think no one wants to read that server telling them and trying to change team again and again, spawning more and more entities with their weapons, wearables and etc until reaching limit. This is why so many entities with same classname such as tf_weapon_bat, of course tf_viewmodel for them and frequently tf_powerup_bottle if was equiped, almost all players trying to pick scout at firts.

You was right, but I even didn't thought on this plugin.

[Kenzzer]

Good to know it's solved.