search

nosoop / SM-TFCWXBaseAttributes

Implementations of Custom Weapons 3 properties as custom attributes.
GNU General Public License v3.0
4 stars 5 forks source link

Server crash by "server_srv.so!CAttributeManager::AddProvider" #9

Closed gaejuck closed 2 years ago

gaejuck commented 2 years ago

[Maintainer note: This was originally posted in the Custom Weapons X issue tracker.]

Environment

Description Sometimes, the server crashes for the following reasons.

https://crash.limetech.org/i6wpz2xd5k5d https://crash.limetech.org/stats/97b6ba06f5099926b9d6be6a4b07035c71efab00febf1afe1496dbbc

If don't use the custom weapon plugin, this won't happen. Also, there are no plugins on the server that affect custom weapon plugin. I don't know the reason why.

nosoop commented 2 years ago

Thanks for the report — can you provide some more info including

The stack trace appears to suggest that indicate that the attribute provider (the owner of the item) isn't valid somehow, but without knowing how the crash happened, that's not much information for me to work with.

gaejuck commented 2 years ago

Here you are :D

  1. Weapon config wep_cfg.zip

  2. Plugins in use

    [SM] Listing 132 plugins:
001 "Simple YouTube Music Player" (1.2.3) by namazso
002 "[TF2] TF2 Attribute Extended Support" (1.4.1) by nosoop
003 "[TF2] Custom Attributes" (0.4.1) by nosoop
004 "TF2 Utils" (0.13.0) by nosoop
005 "Custom Status HUD" (1.1.0) by nosoop
006 "[TF2CA] Banner Buff Override" (1.1.1) by nosoop
007 "cw/ca/cwa_lock_on_projectile.smx"
008 "cw/ca/disable_sniper_unzoom.smx"
009 "cw/ca/cwa_no_scope.smx"
010 "cw/ca/cwa_bonk_ball.smx"
011 "cw/ca/reload_full_clip_at_once.smx"
012 "cw/ca/viewmodel_override.smx"
013 "cw/ca/mod_crit_type_on_hitgroup.smx"
014 "cw/ca/alt_fire_throws_cleaver.smx"
015 "cw/ca/SM-TF2-ProjectileDrop.smx"
016 "[CA] Backpack Object" (1.0) by Foolish
017 "[CA] Human Teleporter" (1.0) by Foolish
018 "cw/ca/cwa_sniper_scope.smx"
019 "cw/ca/cwa_shoot_recoil.smx"
020 "cw/ca/buff_overrides/sniper_rage_buff_reload.smx"
021 "cw/ca/cwa_green_energy_ring.smx"
022 "[TF2] OnTakeDamage Hooks" (1.2.0) by nosoop
023 "Client Preferences" (1.10.0.6512) by AlliedModders LLC
024 "SM File/Folder Downloader and Precacher" (1.5) by SWAT_88
025 "TF Force Holidays" (1.10.2) by Powerlord
026 "Class is dead voice systeam" (1.2) by Classic
027 "SteamWorks Update Check" (1.0b) by Kyle Sanderson
028 "TF2 Full Infinite Ammo" (2.2.6_Final) by Tylerst
029 "TF2: Custom Taunts" (0.1.1) by Batfoxkid
030 "[UMC] Rock The Vote" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
031 "L4D,L4D2,TF2,CS:GO Vote Sniffer" (1.2.2) by Powerlord
032 "sm_rsptc" (0.3) by gameguysz
033 "[UMC] Time Limits" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
034 "Get IP" (0.4) by Firewolf
035 "Updater" (1.2.2) by GoD-Tony
036 "Anti <Unassigned> Stuck" (3.0.0) by Jobggun
037 "Reserved Slots" (1.10.0.6512) by AlliedModders LLC
038 "spray_sound_block.smx"
039 "Discord API" (0.1.106) by Deathknife
040 "asdfasdf.smx"
041 "[UMC] Prefix Exclusion" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
042 "Discord API" (1.0) by .#Zipcore, Credits: Shavit, bara, ImACow and Phire
043 "Discord Utilities" (2.7-BETA) by Cruze
044 "Admin Menu" (1.10.0.6512) by AlliedModders LLC
045 "[UMC] End of Map Vote" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
046 "Third Person" (1.0.8) by Victor Korobkovsky <vitkorob@gmail.com>
047 "Round and Map End Sound" (2.5.0Beta9) by FrozDark (HLModders LLC)
048 "[TF2] Taunt 'em" (1.0.1) by FlaminSarge
049 <Error> "Steph's AntiCheat [StAC]" (5.2.0) by https://sappho.io
050 "[Any] Deluxe Godmode" (2.3.1) by DarthNinja
051 "Snake" (1.0) by Jannik "Peace-Maker" Hartung
052 "Tidy Chat" (0.5) by linux_lover
053 "Teleport Player" (1.8) by Headline, Original: HyperKiLLeR
054 "Advertisements" (0.6) by Tsunami
055 "[TF2] Econ Data Compatibility Layer for TF2II and TF2IDB" (0.7.2) by nosoop
056 "Spray Tracker" (1.11) by Geit
057 "Security Entity Limit!" (0.5.5) by Benoist3012
058 "[UMC] Admin Menu" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
059 "Automatic Steam Update (SteamWorks)" (1.1) by Dr. McKay, Sarabveer(VEER™)
060 "CallAdmin" (0.1.8) by Impact, dordnung
061 "[UMC] Player Count Monitor" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
062 "SourceBans++: SourceComms" (1.6.4) by Alex, SourceBans++ Dev Team
063 "Observer Point" (1.0.100) by <eVa>Dog
064 "AFK Manager" (4.3.0) by Rothgar
065 "Spray Manager" (1.4.2) by Monster Killer
066 "SourceBans++: Bans Checker" (1.6.4) by psychonic, Ca$h Munny, SourceBans++ Dev Team
067 "Tetris" (1.1) by Jannik "Peace-Maker" Hartung
068 "Chat between other servers" (1.8) by ABCDE & Trostal
069 "Map configs" (1.3) by Berni
070 "Melee" (0.7) by linux_lover
071 "quiz_millionaire Custom Quiz Enhancer" (1.01) by Yuumi-Peusah
072 "Websocket" (1.1) by Jannik "Peace-Maker" Hartung
073 "CallAdmin UserManager" (0.1.8) by dordnung, Impact
074 "TF2 Class Restrictions" (0.6) by Tsunami
075 "SuperLogs: TF2" (2.0.32) by Thomas "CmptrWz" Berezansky & psychonic
076 "Fun Votes" (1.10.0.6512) by AlliedModders LLC
077 "TF2 Build Restrictions" (0.5.5) by Tsunami
078 "[UMC] Player Limits" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
079 "SourceBans++: Main Plugin" (1.6.4) by SourceBans Development Team, SourceBans++ Dev Team
080 "Fun Commands" (1.10.0.6512) by AlliedModders LLC
081 "Client Execute" (1.0.0.3) by dubbeh
082 "[UMC] Random Cycle" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
083 "NativeVotes" (1.1.1fix) by Powerlord
084 "TF2 Set Speed" (1.4.0) by Tylerst
085 "Basic Info Triggers" (1.10.0.6512) by AlliedModders LLC
086 "[TF2] STT MultiMod Fix" (1.0.0) by JPZV
087 "Super Spray Handler" (1.3.4) by shavit, Nican132, CptMoore, Lebson506th, and TheWreckingCrew6
088 "Source Chat Relay" (v2.1.5-2-gd81f03c) by Fishy
089 "OldDisconnectMessage.smx"
090 "dust_respawn.smx"
091 "Set Health" (1.3) by Born
092 "[UMC] Post-Played Exclusion" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
093 "Chat Logging" (2.3+1) by R1KO
094 "[TF2] Rate of Fire" (1.2) by EHG
095 "Admin File Reader" (1.10.0.6512) by AlliedModders LLC
096 "SourceBans++: SourceSleuth" (1.6.4) by ecca, SourceBans++ Dev Team
097 "[UMC] Native Voting" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
098 "Anti-Flood" (1.10.0.6512) by AlliedModders LLC
099 "HLstatsX CE Ingame Plugin" (1.6.19) by psychonic
100 "Admin Help" (1.10.0.6512) by AlliedModders LLC
101 "[TF2] TF2Attributes" (1.3.3@nosoop-1.7.0) by FlaminSarge
102 "[ANY] Spin my screen" (1.1) by Tak (Chaosxk)
103 "TF2 Round Clock" (1.0) by Yuumi-Peusah / Foolish
104 "Sound Commands" (1.10.0.6512) by AlliedModders LLC
105 "TF2 Party Korean Patcher, Enhancer" (1.02) by Yuumi-Peusa
106 "socket example" (1.1.0) by Player
107 "TF2 Party Korean Patcher, Enhancer (a12 edition)" (1.01) by Yuumi-Peusa
108 "[UMC] Map Commands" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
109 "[TF2] Wearable Item Tools" (1.1.1) by Powerlord
110 "Basic Votes" (1.10.0.6512) by AlliedModders LLC
111 "Player Commands" (1.10.0.6512) by AlliedModders LLC
112 "[UMC] End of Map Vote Warnings" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
113 "Basic Comm Control" (1.10.0.6512) by AlliedModders LLC
114 "Basic Commands" (1.10.0.6512) by AlliedModders LLC
115 "[UMC] Vote Command" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
116 "[UMC] Echo Nextmap" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence
117 "SourceBans++ Report Plugin" (1.6.4) by RumbleFrog, SourceBans++ Dev Team
118 "Basic Chat" (1.10.0.6512) by AlliedModders LLC
119 "SourceBans++: Admin Config Loader" (1.6.4) by AlliedModders LLC, SourceBans++ Dev Team
120 "LAZERRRRSSSS!" (1.3.2) by MitchDizzle_
121 "SuperLogs: Generic" (1.0) by psychonic
122 "Mann vs. Mann" (1.3.1) by Mikusch
123 "[TF2] Econ Data" (0.17.1) by nosoop
124 "[UMC] Ultimate Mapchooser Core" (3.7.1) by Previous:Steell,Powerlord - Current: Mr.Silence (Foolish Edit)
125 "Foolish Map Enhancer" (0.00) by Yuumi-Peusah
126 "Smug dance" (0.01) by Moonly Days
127 "[TF2] Serverside Player Attachment Fixer" (1.1) by NiagaraDryGuy
128 "Dispenser Here" (1.2.0) by svaugrasn
129 "[TF2] Custom Weapons X" (X.0.8-5abca5d) by nosoop
130 "[TF2] Custom Weapons X - Equip Commands" (1.0.0) by nosoop
131 <Bad Load> player_analytics.smx
132 "Roll The Dice (Revamped)" (2.3.3) by Phil25
Errors:
stac.smx (Steph's AntiCheat [StAC]): Error detected in plugin startup (see error logs)
player_analytics.smx: Unable to load plugin (bad header)

  3. Plugin names associated with the stack trace

    {
    "instruction": 3612157474,
    "module": {
      "base_address": 3611525120,
      "code_file": "jit_code_3611525120_1048576",
      "debug_file": "",
      "debug_identifier": "",
      "size": 1048576
    },
    "plugin": {
      "file": "cw/tf2utils.smx",
      "function": "Native_EquipPlayerWearable"
    },
    "registers": {
      "ebp": 4294705496,
      "eip": 3612157475,
      "esp": 4294705456
    },
    "rendered": "jit_code_3611525120_1048576 + 0x9a623 [ cw/tf2utils.smx::Native_EquipPlayerWearable ]",
    "return_address": 3612157475,
    "stack": "UEluEXD06hHgFwAAIQAAANAAAAADAAAAWAH8/yMmTdckKQAAAgAAAHgB/P9vkPro",
    "trust": 3
  },
  {
    "instruction": 3908741722,
    "module": {
      "base_address": 3908734976,
      "code_file": "jit_code_3908734976_1048576",
      "debug_file": "",
      "debug_identifier": "",
      "size": 1048576
    },
    "plugin": {
      "file": "cw/ca/viewmodel_override.smx",
      "function": ".6832.UpdateClientWeaponModel"
    },
    "registers": {
      "ebp": 4294707240,
      "eip": 3908741723,
      "esp": 4294707168
    },
    "rendered": "jit_code_3908734976_1048576 + 0x1a5b [ cw/ca/viewmodel_override.smx::.6832.UpdateClientWeaponModel ]",
    "return_address": 3908741723,
    "stack": "kFLuEYh38RFADusRABDxEQAQ8RGIZwAAoHvxEa41SNeQUu4RiHfxEQgWAAD4EwAA2AAAAAMAAAAoCPz/rjVI17AaAAACAAAAOAj8/80iSNc=",
    "trust": 3
  },
  {
    "instruction": 3611828940,
    "module": {
      "base_address": 3611525120,
      "code_file": "jit_code_3611525120_1048576",
      "debug_file": "",
      "debug_identifier": "",
      "size": 1048576
    },
    "plugin": {
      "file": "cw/ca/viewmodel_override.smx",
      "function": ".6416.OnInventoryAppliedPre"
    },
    "registers": {
      "ebp": 4294707256,
      "eip": 3611828941,
      "esp": 4294707248
    },
    "rendered": "jit_code_3611525120_1048576 + 0x4a2cd [ cw/ca/viewmodel_override.smx::.6416.OnInventoryAppliedPre ]",
    "return_address": 3611828941,
    "stack": "EBkAAAIAAABYCPz/b5D66A==",
    "trust": 3
  },
  {
    "instruction": 3462309573,
    "plugin": {
      "file": "cw/cwx.smx",
      "function": ".36316.OnLoadoutRespawnPreference"
    },
    "registers": {
      "ebp": 4294711288,
      "eip": 3462309574,
      "esp": 4294711248
    },
    "rendered": "0xce5ea6c6 [ cw/cwx.smx::.36316.OnLoadoutRespawnPreference ]",
    "return_address": 3462309574,
    "stack": "QKJ0KXhzjM34wQkANMMTANAEAAADAAAA+Bf8/8amXs7cjQAAAgAAABgY/P9vkPro",
    "trust": 3
  },
nosoop commented 2 years ago

Hm, that's still strange. The stack trace looks like it's an issue related to some call involving TF2Util_EquipPlayerWearable within the viewmodel attribute, but it's still unclear how CAttributeManager::AddProvider isn't getting a valid interface — it should have a valid owner entity throughout those calls, so it sounds like the provider interface isn't valid.

What's the error at the top of the raw view (on the first tab)? It should start with SIGSEGV or something similar.

Moving this issue to nosoop/SM-TFCWXBaseAttributes since it seems to be an issue within that project.

gaejuck commented 2 years ago

I checked and it says SIGSEGV /SEGV_MAPERR accessing 0x0.

nosoop commented 2 years ago

For debugging purposes, try the attached version. It should log an error if it attempts to run on a valid client without a valid attribute provider interface instead of outright crashing, but it could just crash elsewhere since this change doesn't prevent equipping from other plugins.

You'll probably need to do some more investigating on your own to track down the exact root cause.

hasattributes-fail-test.zip

gaejuck commented 2 years ago

I applied it to the server. and I'll let you know if I get an error log.

gaejuck commented 2 years ago

The log didn't appear, but I think I found the cause. I think the server might crash while equip a custom weapon in a state where there is not enough space for the entity. But I can't figure out why I'm running out of entity space.

L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL]The server is close of the limit, cleanning mode enabled
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL] Entity 1834(tf_wearable) has been deleted
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL] Entity 1835(tf_wearable_vm) has been deleted
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL] Entity 1836(tf_wearable_vm) has been deleted
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL] Entity 1837(tf_wearable) has been deleted
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL] Entity 1838(tf_wearable_vm) has been deleted
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL] Entity 1839(tf_wearable_vm) has been deleted
[snip]
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL]!!!WARNING!!! Server reached the critical limit, SEL is now blocking every entities from spawning.
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL] Entity 1834(tf_wearable) has been deleted
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL] Entity 1835(tf_wearable_vm) has been deleted
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL] Entity 1836(tf_wearable_vm) has been deleted
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL] Entity 1837(tf_wearable) has been deleted
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL] Entity 1838(tf_wearable_vm) has been deleted
L 11/18/2021 - 16:39:52: [security_entity_limit.smx] [SEL] Entity 1839(tf_wearable_vm) has been deleted
[snip]
nosoop commented 2 years ago

That's probably the cause of it. Most of the code runs on the assumption that the wearable is valid once it's spawned (there are validity checks in place in TF2Util_EquipPlayerWearable, but I don't think entities pending deletion are caught by that).

Attributes are initialized in CEconWearable::Spawn, so blocking the spawn process would very likely cause issues with things that use attributes. It looks like that plugin does prevent spawn processes from running.

Closing this issue since there's nothing for me to take action on here.

gaejuck commented 2 years ago

So do you mean it need to disable security_entity_limit.smx to use custom weapons? Rather, prevention is poison...

nosoop commented 2 years ago

security_entity_limit.smx is the direct cause of the server crashing when TF2Utils equips an unspawned cosmetic. The custom weapon model attribute is working as intended up until security_entity_limit.smx starts taking effect.

Provided that's disabled, you'll probably want to figure out why you're hitting the edict / entity limit threshold. Neither CWX nor the clientmodel override attribute should be creating / spawning unnecessary amounts of entities (I estimate it to be +0 for CWX due to the persistence implementation, and +2 per visible weapon for the viewmodel plugin; holstered weapons don't have associated custom model entities).

nosoop commented 2 years ago

nosoop/SM-TFUtils@be04177 should prevent full crashes in the future; instead it'll throw an error indicating that the wearable wasn't initialized / spawned.

gaejuck commented 2 years ago

Server crashes caused by this problem have definitely disappeared. thank you! Now, server crashes caused by ED_Alloc: no free dicts should be resolved.

https://crash.limetech.org/pgrlocb3wn52

"crash_reason": "SIGSEGV /SEGV_MAPERR"

      {
        "instruction": 3762578895,
        "plugin": {
          "file": "cw/ca/viewmodel_override.smx",
          "function": ".11852.TF2_SpawnWearableViewmodel"
        },
        "registers": {
          "ebp": 4292011720,
          "eip": 3762578896,
          "esp": 4292011680
        },
        "rendered": "0xe04465d0 [ cw/ca/viewmodel_override.smx::.11852.TF2_SpawnWearableViewmodel ]",
        "return_address": 3762578896,
        "stack": "AJvkJ1Rh+zN4FgAAdGoAAHgAAAADAAAAyObS/9BlROBMLgAAAgAAANjm0v/UkkPg",
        "trust": 3
      },
  {
        "instruction": 3762524883,
        "plugin": {
          "file": "cw/ca/viewmodel_override.smx",
          "function": ".6832.UpdateClientWeaponModel"
        },
        "registers": {
          "ebp": 4292011736,
          "eip": 3762524884,
          "esp": 4292011728
        },
        "rendered": "0xe04392d4 [ cw/ca/viewmodel_override.smx::.6832.UpdateClientWeaponModel ]",
        "return_address": 3762524884,
        "stack": "sBoAAAIAAADo5tL/fYtD4A==",
        "trust": 3
      },
      {
        "instruction": 3762523004,
        "plugin": {
          "file": "cw/ca/viewmodel_override.smx",
          "function": ".6416.OnInventoryAppliedPre"
        },
        "registers": {
          "ebp": 4292011752,
          "eip": 3762523005,
          "esp": 4292011744
        },
        "rendered": "0xe0438b7d [ cw/ca/viewmodel_override.smx::.6416.OnInventoryAppliedPre ]",
        "return_address": 3762523005,
        "stack": "EBkAAAIAAAAI59L/b4D26A==",
        "trust": 3
      },

I'll investigate further what the problem is and let you know when I get more information.

nosoop commented 2 years ago

You may want to try the Entity Limits Logger plugin to see if you have an excessive amount of certain entities. Not much I can do unless one of my plugins is the cause of that.

gaejuck commented 2 years ago

This is the entity limit log that came out before the server crashed. sm_entity_limit_unsafe_left was set to 500.

entity_limit_2021-11-21_21-58-29.log

nosoop commented 2 years ago

1500 entities is a reasonable amount that a server would normally consume, especially with an increased maxplayers. You may want to configure it to report with 100 entities remaining.

In any case, it doesn't look like there's an excessive number of tf_wearable / tf_wearable_vm entities listed (once the ones created by the game are taken into account). Resolving the issues with entities is outside the scope of the bug report, so I don't think there's anything else I can do for you here, sorry.

gaejuck commented 2 years ago

Sorry to keep fetching logs, but please take a look at this one last time. Count of tf_wearable_vm, tf_wearable is very high compared to other entities. Also I set sm_entity_limit_unsafe_left to 150. Is it normal for this count to come out? If this is a normal situation, what should I do? I don't know what's the cause.

DELTA - {Class Count}
*
    This section describes the number of entity classes, which increased since the latest snapshot
*

+6  tf_weapon_crossbow
+4  tf_weapon_flaregun
+1  tf_weapon_parachute_primary
+12 tf_weapon_shovel
+2  tf_weapon_fists
+2  tf_weapon_pda_spy
+6  tf_weapon_pipebomblauncher
+2  tf_weapon_knife
+4  tf_weapon_grenadelauncher
+30 player
+2  tf_weapon_invis
+3  tf_powerup_bottle
+4  tf_weapon_medigun
+4  tf_weapon_flamethrower
+1  tf_weapon_wrench
+3  tf_weapon_bat
+3  tf_weapon_bottle
+1  tf_weapon_stickbomb
+9  tf_weapon_buff_item
+285    tf_wearable_vm
+2  tf_weapon_revolver
+108    tf_viewmodel
+5  tf_weapon_sniperrifle
+1  tf_weapon_soda_popper
+1  tf_weapon_jar
+241    tf_wearable
+3  tf_wearable_razorback
+6  tf_weapon_builder
+1  tf_weapon_handgun_scout_secondary

entity_limit_2021-11-22_20-44-42.log

nosoop commented 2 years ago

That indeed does not appear to be normal.

gaejuck commented 2 years ago

There was no error log. Also, security_entity_limit.smx was already deleted because it was judged to be of little help to the server. And these models are not used for anything other than just overriding on weapon model.

nosoop commented 2 years ago

Are the weapons using those models being spawned via loadout changes on CWX, or granted via sm_cwx_equip_target (or some other plugin)? I don't think it'd make a difference either way (from my own tests), but good to double-check on that.

Stumped here 🙁

gaejuck commented 2 years ago

sm_cwx_equip_target was not used. Just that users wore weapons through sm_cwx.