Error

[GoogleCodeExporter]

What steps will reproduce the problem?
1.  Running the server after a long period of time

What is the expected output? What do you see instead?
I don't expect the server to crash.

What version of the product are you using? On what operating system?
Latest version, Windows 2003

Please provide any additional information below.
2:48 PM - Rigby: threadtools.cpp <2574> : Assertion Failed: Illegal termination 
of worker thread ' CN et Encrypt:0'
2:49 PM - Rigby: Assert( Assertion Failed: Illegal termination of worker thread 
' CNet En crypt:0'):.\threadtools.cpp2574

Original issue reported on code.google.com by erie1555 on 22 Nov 2011 at 7:56

[GoogleCodeExporter]

What module is this bug report for?

Original comment by AzuiSl...@gmail.com on 22 Nov 2011 at 7:58

[GoogleCodeExporter]

I'm not completely sure but I was told it was a GarrysMod module

Original comment by erie1555 on 22 Nov 2011 at 8:08

[GoogleCodeExporter]

If you don't know what is causing the crashes, you'll need to disable addons 
such as metamod and sourcemod as well as any modules in your 
/lua/include/modules/ until you identify which one is causing the crashes.

Original comment by AzuiSl...@gmail.com on 22 Nov 2011 at 8:14

[GoogleCodeExporter]

Also getting this.

I don't run metamod or sourcemod, and disabled all other modules for testing.

Seems to be caused by tmysql. Happens for me in both tmysql2, and 
tmysql3_boost. It never used to happen until a few engine updates ago.

Original comment by jamesros...@gmail.com on 17 Dec 2011 at 7:27

[GoogleCodeExporter]

What OS? do you have crash dumps? do you have a testcase? what other modules 
are you running?

It's unlikely that the mysql module can cause that kind of crash.

Original comment by AzuiSl...@gmail.com on 17 Dec 2011 at 10:23

[GoogleCodeExporter]

Win server 2003. 

There's no real sure-fire way to reproduce it I've found, as it somewhat 
happens "randomly" after the server has been running for a while. Sometimes 
it'll occur a few hours after I restart my server, and other times, it can take 
a few days.

I usually run sv_geoip, mysqloo, and tmysql. I spent some time trying to 
pinpoint which one was causing it, and running each module individually for 
multiple days, the only one to cause any issue was tmysql.

Here's a dump (be it a slightly old one where I ran both tmysql and mysqloo): 
http://dl.dropbox.com/u/3287852/crash_srcds_rp.exe_20111121185748_1.dmp

Original comment by jamesros...@gmail.com on 18 Dec 2011 at 4:51

[GoogleCodeExporter]

Looking at that crash dump, none of the tmysql threads have crashed. The 
exception in that crash dump points to vphysics.

Original comment by AzuiSl...@gmail.com on 18 Dec 2011 at 5:29

[GoogleCodeExporter]

Here's another one that happened just recently. The error mentioned above hung 
on my srcsd.exe until I forcefully terminated it.

http://dl.dropbox.com/u/3287852/crash_srcds.exe_20111218170426_1.dmp

Original comment by jamesros...@gmail.com on 18 Dec 2011 at 5:32

[GoogleCodeExporter]

And another: 
http://dl.dropbox.com/u/3287852/crash_srcds.exe_20111218182925_1.dmp

Original comment by jamesros...@gmail.com on 18 Dec 2011 at 7:19

[GoogleCodeExporter]

None of these crash reports seem to point to gm_tmysql. Perhaps you should try 
-verify_all your server files to make sure you're not using any outdated bins?

Original comment by AzuiSl...@gmail.com on 19 Dec 2011 at 12:16

[GoogleCodeExporter]

Done so, many a time, and I can confirm that when running the server without 
tmysql, these crashes do not occur.

Original comment by jamesros...@gmail.com on 19 Dec 2011 at 12:27

[GoogleCodeExporter]

Looking at your crash dumps, you aren't using a stock vphysics.dll, so I would 
suggest checking that you are using the latest binaries.

The stack traces do not point to gm_tmysql and I can't see any modules on the 
stack that would indicate that it was the issue. The only way I can help you is 
if you can isolate it to a test case.

If you're not already use tmysql3 or tmysql3_boost I would recommend trying one 
of them incase there is an issue in tmysql2, but the only difference is really 
in the shutdown behavior.

Original comment by AzuiSl...@gmail.com on 19 Dec 2011 at 12:35

[GoogleCodeExporter]

Very strange. I'll try removing vphysiscs.dll and downloading it again.

I am already using tmysql3_boost, but I found this to occur with both versions, 
as I mentioned above.

Original comment by jamesros...@gmail.com on 19 Dec 2011 at 12:46

[GoogleCodeExporter]

Actually, I should correct what I said:

It seems like none of your orangebox/bin modules are up to date, are you sure 
you've done a -verify_all?

Original comment by AzuiSl...@gmail.com on 19 Dec 2011 at 12:51

[GoogleCodeExporter]

Updated for latest engine binaries.

Will report back if/when the error happens again.

Original comment by jamesros...@gmail.com on 19 Dec 2011 at 1:43

[GoogleCodeExporter]

Crash happened again.
Srcds hang with the error message until I forcefully closed it:
http://dl.dropbox.com/u/3287852/crash_srcds.exe_20111219090657_1.dmp

Original comment by jamesros...@gmail.com on 19 Dec 2011 at 5:57

[GoogleCodeExporter]

And another: 
http://dl.dropbox.com/u/3287852/crash_srcds.exe_20111219224458_1.dmp

Let me know if I should stop posting crash dumps. Just trying to be as helpful 
as I can.

Original comment by jamesros...@gmail.com on 19 Dec 2011 at 10:47

[GoogleCodeExporter]

These crash dumps all point to the same function, which indicates a gamemode or 
map issue. If I can I'll do some analysis on the crash dump.

gm_tmysql does not do any interaction with vphysics, are you sure it's not a 
result of functions you are calling as a result of a query?

Or, if it's actually frozen, you should capture a full memory dump with a tool 
like ProcDump and email it.

Original comment by AzuiSl...@gmail.com on 19 Dec 2011 at 11:04

[GoogleCodeExporter]

Very strange.

I do nothing odd with data received from my mysql tables; just basic 
concommands retrieving data, appending to player ents, and then saving new 
data. Nothing that would interact with vphysics (to my knowledge). 

I'll get ProcDump though and email that dump the next time it occurs. :)

Thanks for all the assistance though. Really appreciate it. :)

Original comment by jamesros...@gmail.com on 19 Dec 2011 at 11:18

[GoogleCodeExporter]

Alright I did some analysis and the dump, are you using ragdolls or vehicles in 
your gamemode?

Original comment by AzuiSl...@gmail.com on 20 Dec 2011 at 1:07

[GoogleCodeExporter]

This happens in all 3 of my servers using tmysql. 
One is a Fretta server, so yes a few gamemodes use vehicles / ragdolls.
Another is an RP, and actually, we don't use vehicles or ragdolls there at all.
The final server is a stronghold one, which also doesn't use either.

The RP server is the one most prone to crashes and will crash with this error 
about 3-4 times a day. The other servers will only crash with this error once 
every few days.

Original comment by jamesros...@gmail.com on 20 Dec 2011 at 1:24

[GoogleCodeExporter]

All of the crash dumps you've given me are for "Game Servers/RP" and all of 
them point to similar functions in vphysics.

If you've tried different versions of the module and it still crashes, that 
means that one suspect module could be libmysql.dll. Do you have the latest 
libmysql.dll for the version of mysqld you're running?

If you have crashes for the other servers I can look at them, but to me it 
looks like a vphysics bug.

Original comment by AzuiSl...@gmail.com on 20 Dec 2011 at 2:15

[GoogleCodeExporter]

libmysql was actually one of my first thoughts. We were running a terribly old 
copy, so I grabbed the latest one from our mysql setup. The issue still 
occurred.

Since we recently moved server boxes, I do not have any other dumps from the 
other servers, but if any do occur, I'll be sure to throw them your way.

Thanks for all the help. :)

Original comment by jamesros...@gmail.com on 20 Dec 2011 at 2:23

[GoogleCodeExporter]

I updated the Shutdown behavior of tmysql3 and tmysql3_boost, in case there 
were some leaks causing issues. I don't know when you get your crashes, but it 
might not help your RP server if it doesn't changelevel.

Original comment by AzuiSl...@gmail.com on 20 Dec 2011 at 7:02

[GoogleCodeExporter]

My RP server doesn't changelevel very often, like you assumed.

Nevertheless though, I'll update to the most recent version, and let you know. 
:)

The crashes seem to happen somewhat randomly. I've never been able to pinpoint 
an exact cause.

Original comment by jamesros...@gmail.com on 20 Dec 2011 at 5:08

[GoogleCodeExporter]

Server was going good for about 24 hours, until just now when it crashed again 
with the same error:
http://dl.dropbox.com/u/3287852/crash_srcds.exe_20111221181800_1.dmp

Original comment by jamesros...@gmail.com on 21 Dec 2011 at 6:37

[GoogleCodeExporter]

Been getting crashes for the last few days still, but I just noticed something 
else that happened right before a crash. The following was spammed quite a few 
times:
05:37:33 Lua Error: Tried to pop off the top ( Stack is 1, tried to pop 2 )!

I can provide more crash dumps if necessary.

Original comment by jamesros...@gmail.com on 28 Dec 2011 at 5:56

[GoogleCodeExporter]

crash_srcds.exe_20111221181800_1.dmp shows what looks like a crash in 
PhysicsEnvironment::CreateFixedConstraint. Are you sure this isn't a gamemode 
issue? or perhaps a hardware issue?

Original comment by AzuiSl...@gmail.com on 28 Dec 2011 at 9:57

[GoogleCodeExporter]

If the issue is corruption, it may be that those console errors you saw are the 
source of the issue. 

I compiled a quick hacked tmysql that will write minidumps when you get a stack 
error ("tried to pop off the top"). It will generate dumps in orangebox\ with 
the name error_%tickcount%_%processid%_%threadid%.dmp.

http://dl.dropbox.com/u/759758/gm_tmysql_errorintercept.zip

If you can send me one of those I might be able to diagnose the issue further, 
otherwise all I have to work with are the possibly corrupted dumps pointing to 
vphysics.

Original comment by AzuiSl...@gmail.com on 28 Dec 2011 at 12:10

[GoogleCodeExporter]

I highly doubt it's a hardware issue, as this has happened on both my old 
server box, and my brand new one. It also occurs on all 3 of my servers (be it 
less on the other two though), so a gamemode issue is also unlikely. The crash 
seems to disappear if I remove tmysql.

I'll run that tmysql version though, and get back to you with any of those 
dumps. Thanks again for all your help. :)

Original comment by jamesros...@gmail.com on 28 Dec 2011 at 4:12

[GoogleCodeExporter]

Alright, I did some more investigation. It could be a "refcount death spiral." 
I've updated both modules to fix that,

http://code.google.com/p/gmodmodules/source/detail?r=176

If you have any dumps send them to me and I'll check them out, otherwise give 
this newest version a shot. I have a feeling this is the source of the issue.

Original comment by AzuiSl...@gmail.com on 29 Dec 2011 at 11:38

[GoogleCodeExporter]

Alright thanks. Unfortunately my server hasn't crashed since you gave me that 
version above, so I can't provide any more dumps from that.

I'll give this a shot though, thanks. :)

Original comment by jamesros...@gmail.com on 29 Dec 2011 at 2:00

[GoogleCodeExporter]

After about 30 hours or so of running flawlessly, the same error occurred. :(

http://dl.dropbox.com/u/3287852/crash_srcds.exe_20111230194446_2.dmp

Original comment by jamesros...@gmail.com on 30 Dec 2011 at 7:47

[GoogleCodeExporter]

Here's the module compiled with the debug code again:

http://dl.dropbox.com/u/759758/gm_tmysql_errorintercept2.zip

Check your orangebox\ folder just incase it wrote a dump before and didn't 
crash the server.

Original comment by AzuiSl...@gmail.com on 31 Dec 2011 at 1:53

[GoogleCodeExporter]

I'll run that again and let you know of anything. :)

The only other dump file I have here not prefixed with crash_ is an assert_ one:

http://dl.dropbox.com/u/3287852/assert_srcds.exe_20111230045606_1.dmp

Original comment by jamesros...@gmail.com on 31 Dec 2011 at 2:08

[GoogleCodeExporter]

Just had another crash. Only dump was 
http://dl.dropbox.com/u/3287852/crash_srcds.exe_20111231043430_1.dmp

Original comment by jamesros...@gmail.com on 31 Dec 2011 at 4:38

[GoogleCodeExporter]

This one appears to have the same stack trace as the previous crash dumps. It 
looks like you're running the debug module. Just to make sure: the module 
writes dumps to the orangebox\ root, not the dumps sub-folder.

If there aren't any errors_*.dmp, then I'm going to investigate the vphysics 
crash, since it's happening so frequently.

Original comment by AzuiSl...@gmail.com on 31 Dec 2011 at 5:17

[GoogleCodeExporter]

I'm running the errorintercept version you linked above, yes, and I have no 
dumps in orangebox/ root unfortunately, sorry.

So grateful for all your help so far though. Thanks! :)

Original comment by jamesros...@gmail.com on 31 Dec 2011 at 5:31

[GoogleCodeExporter]

There's a possibility the vphysics crash is coming from a constraint setup 
after a call to Entity:Activate.

Could you add something like

local EntityMeta = _R.Entity;

local EntityActivate = EntityMeta.Activate;

function EntityMeta:Activate()
    filex.Append("activation_log.txt", "Activating " .. self:GetClass() .. "\n");
    EntityActivate(self);
end

to a script in autorun/server. If it is a vphysics issue we should see the 
entity before it crashes. We can continue from there.

Original comment by AzuiSl...@gmail.com on 31 Dec 2011 at 8:44

[GoogleCodeExporter]

Will do. :)

I'll get back to you with that file after the next crash, thanks. :)

Original comment by jamesros...@gmail.com on 31 Dec 2011 at 3:54

[GoogleCodeExporter]

Just crashed again.
Console screenshot:
http://puu.sh/cePQ

Last few lines of activation log:
Activating prop_physics
Activating phys_constraintsystem
Activating phys_hinge
Activating phys_torque

Found this is orangebox/: 
http://dl.dropbox.com/u/3287852/error_1737205671_102420_321680.dmp however 
strangely, there was no actual crash dump generated this time.

Original comment by jamesros...@gmail.com on 2 Jan 2012 at 10:24

[GoogleCodeExporter]

A few more crash dumps, for what it's worth:
http://dl.dropbox.com/u/3287852/crash_srcds.exe_20120105223159_1.dmp
http://dl.dropbox.com/u/3287852/crash_srcds.exe_20120106000802_1.dmp

And before the most recent one:
Activating phys_ragdollconstraint
Activating phys_ragdollconstraint
Activating phys_ragdollconstraint
Activating phys_ragdollconstraint

Original comment by jamesros...@gmail.com on 6 Jan 2012 at 12:59

[GoogleCodeExporter]

I meant to comment on this one earlier. This is likely a wider vphysics issue 
rather than any gmod module. I've posted on the HLDS mailing list as I have 
similar issues with TF2 where the server and client crash in vphysics.

I'm confident that any issues that existed in gm_tmysql are fixed now, and the 
remaining issue is simply vphysics.

If for some reason the bug is happening more frequently than you'd like, you 
may want to prevent phys_ragdollconstraints or perhaps all constraints from 
being created.

Original comment by AzuiSl...@gmail.com on 6 Jan 2012 at 2:25

[GoogleCodeExporter]

Alright, thanks for the response.

Appreciate all the help. :)

Original comment by jamesros...@gmail.com on 6 Jan 2012 at 2:40

[GoogleCodeExporter]

If you're still interested in helping me diagnose the issue, I'd like to see if 
you can find a "stable" version of vphysics from the previous versions.

http://dl.dropbox.com/u/759758/313_137_to_145.zip

I would start with 137 and work up to 141 to see if any of those still crash. 
145 is the current version of vphysics.

Original comment by AzuiSl...@gmail.com on 7 Jan 2012 at 12:15

[GoogleCodeExporter]

I'll definitely start doing that later tonight, thanks. :)

Original comment by jamesros...@gmail.com on 7 Jan 2012 at 12:24

[GoogleCodeExporter]

Whilst running 137, the crash still occurred, with the following:
http://puu.sh/cH2j

Should I continue to increase the version used, or perhaps go earlier?

Original comment by jamesros...@gmail.com on 7 Jan 2012 at 5:41

[GoogleCodeExporter]

Inconclusive, may be related to Issue #19

Original comment by AzuiSl...@gmail.com on 3 Feb 2012 at 12:18

• Changed state: WontFix