player_disconnect called from CParallelProcessor crashes on CLuaGamemode::CallWithArgs with !ThreadInMainThread

[Python1320]

Billyboy17: lag
Giving Охотниk a weapon_archerbow
Bad SetLocalOrigin(-15047.433594,-3763.225586,-27608.164062) on gmod_hands (334)
Giving Охотниk a weapon_archerbow
ERROR! SendData reliabe data too big (538718)[Disconnect] 4 Player [3][Pewdiepie]        (STEAM_0:0:1691126101): ERROR! Couldn't send snapshot
ERROR! SendData reliabe data too big (539567)[GM:CallWithArgs - !ThreadInMainThread] player_disconnect
[GM:CallWithArgs - !ThreadInMainThread] player_disconnect

[GM:CallWithArgs - !ThreadInMainThread] player_disconnect

[CRASH] Not on main thread! 
Signal 11 at fault (nil), caller = 0xf618acd0 with sicode 1 (Address not mapped)

Regs> gs 63 fs: 0 es: 2b ds: 2b edi: edd9f64c esi: 1 ebp: edd9f5e8 esp: edd9f1c0 ebx: edd9f1dc edx: 67 ecx: 0
    > eax: e trap: 6 err: f618acd0 eip: 23 cs: 10246 flag: edd9f1c0 sp: 2b ss: 0 cr2: edd9f1dc

Stack trace:
> 1     [0xda92acc4,0xedd9e880] ERROR_SIGNAL_HANDLER_FUNC(int, siginfo*, void*) +0x794                  @ gmsv_segfault_linux.dll+0x4cc4.
> 2     [0xf775e410,0xedd9ee40] ? +0x794                @ linux-gate.so.1+0x410.
> 3     [0xf618acd0,0xedd9f1c0] Sys_Error_Internal(bool, char const*, char*) +0x160             @ engine_srv.so+0x1b8cd0.
> 4     [0xf618ad90,0xedd9f5f0] Sys_Error(char const*, ...) +0x20               @ engine_srv.so+0x1b8d90.
> 5     [0xf618c0f8,0xedd9f610] Sys_SpewFunc(SpewType_t, char const*) +0x2d8            @ engine_srv.so+0x1ba0f8.
Dropped Pewdiepie from server (ERROR! Couldn't send snapshot.)
> 6  [0xe985c74d,0xedda1670] newspew(SpewType_t, char const*) +0x27d                 @ gmsv_xterm_linux.dll+0x774d.
> 7     [0xf741cddf,0xedda16d0] _SpewMessage(SpewType_t, char const*, int, Color const*, char const*, char*) +0xbf              @ libtier0_srv.so+0xeddf.
> 8     [0xf741d82c,0xedda2ae0] Error +0x2c             @ libtier0_srv.so+0xf82c.
> 9     [0xee5d2fe8,0xedda2b00] CLuaGamemode::CallWithArgs(char const*) +0x78           @ server_srv.so+0x5d2fe8.
> 10    [0xee5f6fce,0xedda2b20] CLuaGameEventListener::FireGameEvent(IGameEvent*) +0x13e                @ server_srv.so+0x5f6fce.
> 11    [0xf60dcad6,0xedda2b90] CGameEventManager::FireEventIntern(IGameEvent*, bool, bool) +0xd6               @ engine_srv.so+0x10aad6.
> 12    [0xf60dce48,0xedda2be0] CGameEventManager::FireEvent(IGameEvent*, bool) +0x28           @ engine_srv.so+0x10ae48.
> 13    [0xf616387a,0xedda2c00] CGameClient::Disconnect(char const*, ...) +0x15a                @ engine_srv.so+0x19187a.
> 14    [0xf608cf18,0xedda3050] CBaseClient::SendSnapshot(CClientFrame*) +0x578                 @ engine_srv.so+0xbaf18.
> 15    [0xf6165214,0xedda30e0] CGameClient::SendSnapshot(CClientFrame*) +0x1c4                 @ engine_srv.so+0x193214.
> 16    [0xf6175c4f,0xedda3180] SV_ParallelSendSnapshot(CGameClient*&) +0x3f            @ engine_srv.so+0x1a3c4f.
> 17    [0xf617956a,0xedda31a0] CParallelProcessor<CGameClient*, CFuncJobItemProcessor<CGameClient*> >::DoExecute() +0x3a               @ engine_srv.so+0x1a756a.
> 18    [0xf732c307,0xedda31d0] CFunctorJob::DoExecute() +0x17                  @ libvstdlib_srv.so+0x12307.
> 19    [0xf732ce6f,0xedda31f0] CJob::Execute() +0x7f           @ libvstdlib_srv.so+0x12e6f.
> 20    [0xf732db08,0xedda3220] CJobThread::Run() +0x358                @ libvstdlib_srv.so+0x13b08.
> 21    [0xf741f578,0xedda3300] CThread::ThreadProc(void*) +0x88                @ libtier0_srv.so+0x11578.
> 22    [0xf760e00b,0xedda3330] start_thread +0xdb              @ libpthread.so.0+0x700b.
> 23    [0xf754899e,0xedda3430] clone +0x5e             @ libc.so.6+0xf399e.
END: Unspecified (general) error

Lua Trace:

STACK (0): .
Unloading crashhandlers... calling previous crash handler

[guillaume-dorczynski]

Same error crashing server

DataTable warning: player: Out-of-range value (324.916382) in SendPropFloat 'm_angEyeAngles[0]', clamping.
DataTable warning: player: Out-of-range value (324.916382) in SendPropFloat 'm_angEyeAngles[0]', clamping.
DataTable warning: player: Out-of-range value (324.916382) in SendPropFloat 'm_angEyeAngles[0]', clamping.
ERROR! SendData reliabe data too big (21156)[GM:CallWithArgs - !ThreadInMainThread] PlayerDisconnected

[mleonhard38]

ERROR! SendData reliabe data too big (110792)[GM:CallWithArgs - !ThreadInMainThread] PlayerDisconnected
[GM:CallWithArgs - !ThreadInMainThread] PlayerDisconnected

[GM:CallWithArgs - !ThreadInMainThread] PlayerDisconnected

./srcds_run: Zeile 372: 27227 Speicherzugriffsfehler  $HL_CMD
Add "-debug" to the ./srcds_run command line to generate a debug.log to help with solving this problem
Mo 30. Jun 19:14:20 CEST 2014: Server restart in 10 seconds

This still persists.

[Python1320]

This might also be the reason why PlayerDisconnected isn't always reliable on windows? That is if this crash doesn't happen on windows (and nothing is done).

[meanwhile0]

The crash does happen on Windows. It happens quite frequently on my server.

[willox]

Quoting myself from another issue ( https://github.com/Facepunch/garrysmod-issues/issues/1272 ): "This is caused by a change to PLAYER.Kick and PLAYER.Ban. They now perform their actions instantly rather than when the command-buffer is executed and cause all sorts of problems with different mods."

Could this be related?

[Bo98]

This issue was opened in January and the Kick/Ban change happened afterwards.

[willox]

Well spotted oops

Although I think similar issues could occur through it the same way that gatekeeper's kick function used to.

[Mista-Tea]

I've been having this crash occur several times a day now since the past few updates (or at least I've been seeing more of them because serverdoc sometimes fails to close the popup window that opens showing that error). Windows 2012 server.

http://facepunch.com/showthread.php?t=1348923&p=45390641&viewfull=1#post45390641 http://facepunch.com/showthread.php?t=1374457&p=45441137&viewfull=1#post45441137

[Mista-Tea]

It looks like this issue might be related/escalated to DataTable being messed up in the last update(s): http://facepunch.com/showthread.php?t=1374457&p=45575254&viewfull=1#post45575254

For me, I was using SetDT/GetGet with players. Changing those to SetNW/GetNW for the meantime has fixed the crazy lag issue, and possibly the player_disconnect crash. I'm basing this off of xaviergmail's post, since we both had the same issue and removing DT shit fixed it for him.

[mleonhard38]

ERROR! SendData reliabe data too big (136112)
[GM:CallWithArgs - !ThreadInMainThread] player_disconnect
[GM:CallWithArgs - !ThreadInMainThread] player_disconnect
[GM:CallWithArgs - !ThreadInMainThread] player_disconnect
Segmentation fault

Still relevant. @Mista-Tea seems it doesnt make a difference tbh.

[meepen]

Bump, it still happens

http://a.pomf.se/mtxled.mdmp

[Lumixz]

Can confirm this is still a problem. Happens while players are joining in after a map change, before the game even fully starts. Happens around 4 times a day. Windows machine.

I don't have a dump, just popping in to bring attention to this.

[drizmans]

Still an issue. Happens when only under a high load of players ( About 50-60+ ) and seemingly random. Linux machine, just to throw a spin on things.

ERROR! SendData reliabe data too big (153349)[GM:CallWithArgs - !ThreadInMainThread] >PlayerDisconnected [GM:CallWithArgs - !ThreadInMainThread] PlayerDisconnected

[GM:CallWithArgs - !ThreadInMainThread] PlayerDisconnected

[robotboy655]

It looks like it crashes when trying to kick a player after failing to send a full snapshot update or some shit.

[robotboy655]

Yeah, I am pretty sure that's what happens. The player is supposed to be kicked with the reason "ERROR! Couldn't send snapshot.".

[robotboy655]

I believe you should be able to fix it for your server with sv_parallel_sendsnapshot 0. I don't know what the drawbacks are, but hey, it should fix this crash.

[Lumixz]

I'll try it out. Thanks.

[drizmans]

I've just done it so I'll let you know. Cheers mate.

[drizmans]

Crashed.

ERROR! SendData reliabe data too big (156742)Dropped A Sneaky Panda from server (ERROR! Couldn't send snapshot.) rip community.

[Lumixz]

Check your .cfg again. I noticed after I changed mine, it reverted for some reason.

[drizmans]

It is in my server.cfg file. I ran it through console first, then it continued to crash with a different message. ( More detailed this time ) Now it booted with the cvar it may make a difference - no idea with this game...

[drizmans]

Sorry, can I clarify - did you mean it set back during the servers runtime, or after a crash it hadn't set? If the case is the former I could make a timer to check the cvar I guess...

[Lumixz]

For me, it changed back while running. I'm not sure if it was that it never actually saved when I thought I had done it, if my host's managed system did it, or something else. Probably just didn't save. However i suspect your server needs to boot with it set for it to take effect.

[drizmans]

Okay, thankyou - I'll be sure to post back with any further developments/failures!

[drizmans]

Inside fountain :< ERROR! SendData reliabe data too big (153729)Dropped TragicChicken from server (ERROR! >Couldn't send snapshot.) Segmentation fault

Nope. Not fixed. I hope they fix this soon - its destroying my community and really stressing me out. We are struggling to raise enough for the bills because of 4-5 crash's a day.

[robotboy655]

Your crash is different, I am not seeing anything about player_disconnect or PlayerDisconnect in your supposed crash logs.

[drizmans]

Right, is there any information you can give me on that issue as searching the internet for the string "ERROR! SendData reliabe data too big" doesn't show much more than this thread? Anything like if it could be caused by Lua ( Really should Lua be able to crash a server? ) or would it be a library causing it? You know a lot more about the game than most; as such I'm asking for any information you might know. Thankyou.

[robotboy655]

Not without at least the stack trace.

[drizmans]

That gets produced by the debug.log or the crash dumps?

[robotboy655]

I have no idea what it is on Linux.

[drizmans]

As far as I'm aware the debug.log ( Produced by the "-debug" command ) and Garrysmod crash dumps ( Always created upon a crash automatically ) are the same for Windows and Linux. I was asking which would contain the information most useful as I'd like to waste as little of your time as possible. Unless you'd need both? Or am I off the cliff here and the stack trace is something completely different?

[drizmans]

Made a independent Facepunch thread to stop spamming everyone following the post.

[Lexicality]

Memory dumps are always useful for diagnosing crashes. Debug logs sometimes. When in doubt provide all information available.

On Sun, 21 Jun 2015 22:11 SnowboiTheGr8 notifications@github.com wrote:

Should I make a facepunch thread for this?

— Reply to this email directly or view it on GitHub https://github.com/Facepunch/garrysmod-issues/issues/697#issuecomment-113960060 .

[Lumixz]

His server might crash before it fully prints the error. That's what happens to mine, too. The full error I usually get is:

ERROR! SendData reliabe data too big (41816)>"xxxxxxxxx<230><>" disconnected (reason >"ERROR! Couldn't send snapshot.") Dropped xxxxxxxxxxxxxx from server (ERROR! Couldn't send snapshot.) Engine error: [GM:CallWithArgs - >!ThreadInMainThread] player_disconnect

Where the x's are player name or steam ID. Sometimes more than one person can get kicked, with all of those lines repeating for each kick. Then the server dies. Sometimes it dies before it even prints the player getting kicked.

[drizmans]

I can actually vouch, 3 crashes later - they all seem to cut off differently. What I don't get is before I set the cvar to 0 I never got the

Dropped xxxxxxxxxxxxxx from server (ERROR! Couldn't send snapshot.)

Rather what you can see in one of my first posts. Really obscure. :(

[drizmans]

Quick update; just had this

DLL_MessageEnd: Refusing to send user message TextMsg of 256 bytes to client, user message size limit is 255 bytes

Happen at the same time as the crash? Not sure if its related.

Update; Ignore me its related to chat messages.

[Lumixz]

I'm not sure why, but the crashing seems to have stopped for me. Last crash was the 21st. Looking at my changelog, I haven't really done anything significant that I think could have fixed it.

[drizmans]

After contacting Noiwex who I must thank for his information that he shared, it appears the crashing can be caused by too many entities. We've drastically reduced the amount of entities that'd be able to be on the map concurrently.

Lumixz have you recently had significantly less entities on your map?

[Lumixz]

In fact we did make some changes to weapon spawns on some maps around the time the crashing stopped, although I'm pretty sure it has crashed on maps that we did not edit. But that does seem to line up. I'll go through and remove some.

Thanks for the information.

[drizmans]

I've also been told that some weapon bases can cause it. I'm not sure how that even makes sense, really - what weapon base are you using?

[Lumixz]

Default TTT.

[drizmans]

I'm pretty sure TTT uses CSS by default, we use FA:S which seems stable, switched from CW 2.0. Thanks.

[Lumixz]

TTT has its own weapon base.

[Mista-Tea]

I hadn't knowingly had my sandbox server crash from this for quite some time. Unfortunately, it appears to have just happened. I thought I was in the clear but I guess not.

[Lumixz]

I didn't notice this before, but nearly every single time this has happened, the player with the snapshot error belongs to my ULX Mod or Admin groups.

[meepen]

maybe it's trying to send too much data to an entity that doesn't fully exist yet? like via net

[robotboy655]

It's simply sending too much data.

[Lumixz]

Do you think it would be possible to patch the method that it uses to kick players, preferably in a way that isn't broken?

[robotboy655]

It's not as simple as "patch it". We don't even have a 100% way to reproduce the crash on demand.

[Lumixz]

Quick update, player got kicked for the snapshot error but it DID NOT crash the server this time. Again, they were in a ulx/ulib group other than user.