1.8.4 - Client crashes

[GeryonTom]

Okay we've found the cause of the client side crashes. If you get hit by zombies beside buildables eventually your client will crash. It's reproducible every time. This is a serious problem that's been on going and we've finally narrowed it down.

Here's a link to the dml for the buildables on my server. https://www.dropbox....dables.sql?dl=0

Place them in a test server and invite zombie hits and eventually your client crashes. It's possible that one of you has to be the owner

Set your coords to here so that you can be close to one of the bases: 2143.8,6213.18,0.001. I'll be banning and buildings until corrected.

Video on crashes: Around 6:10 is where it crashes https://www.youtube.com/watch?v=ML5XY-SscRs&feature=youtu.be

Around 5:30 https://www.youtube.com/watch?v=sbTS9UwfQww&feature=youtu.be

I've added logs: https://www.dropbox.com/s/60i7ljgm87bmx22/ArmA2OA.mdmp?dl=0 https://www.dropbox.com/s/t4fy2wqs8ufcq10/ArmA2OA.RPT?dl=0

Cheers,

Geryon

[Lukethero]

Thanks for the credit ;D

[Lukethero]

But really, this is a really real problem and will crush servers when people try to base build. This is a need to fix, quickly, before the patch is released.

[Lukethero]

Not exclusive to zombies, I have been hit by bullets a few times and it has crashed, didn't crash everyone else though.

[R4Z0R49]

Base building has been in a few versions now we haven't had any reports of crashing ?

I cant see why anything would be causing a crash built items have been ingame along time and walls don't have any sort of damage system atm.

Cant fix if i don't know whats broke to start with.

1.8.4 has already been submitted its been in BIS hands about a week and a half if not longer now.

[GeryonTom]

I realize you can't do anything for 1.8.4. Can we try reproduce with the DML (inserting building objects) into a test server post 1.8.4? Can we at least try to establish if this is a global issue (or not)? The circumstances to reproduce the issue isn't easy. The walls need to be in an area where zombies spawn and then it might happen if they hit you. Also not very many people really even bother to build (the novelty wore off after 1.8.2 on US 434,435, 436). Thus it's possible the right conditions have not been triggered on other servers. For example we started noticing issues when people built walls in Stary.

Also this is the first time reporting the issue but mass client crashes have been happening for at least several months. Until recently we haven't been able to reliably reproduce. For many months we just assumed it was due to hackers doing client crash scripts.

[ericandersen]

Anecdotally I can confirm seeing these issues, specifcally like Tom says, near Zombie populations in towns with player walls (IE in Stary on 434 is a perfect example).

[R4Z0R49]

I really cant see anything that would be causing this issue so unless someone can feed more info into this not alot i can really do. I don't have time to try and track this down if someone can track down the exact problem and a way to recreate every time ill try and have it fixed.

None of my own community have reported issues like this and we do build near zed spawning areas.

But like i say if someone tracks the exact issue down (if there is one) then ill do what i can to fix it.

[Lukethero]

More than just zombies, but if you do not believe this is a real issue, go to the cords given to you by Tommy and tell me it is not game breaking.

[Lukethero]

Me and Tommy are working on finding the precise cause, but chances are we will need someone who is used to debugging to read into the code once we do. More info to come soon.

[R4Z0R49]

No one is saying they don't believe you.

What i'm saying is narrow it down. If you can narrow it down to be a 100% crash every time on a server that is pure dayz nothing extra including be or filters or any 3rd party system and report that i'll be able to find the exact cause of the issue. But as it is standing next to a build-able and takeing damage is really random. I've tested that already and haven't crashed yet even left my client running for 30 mins while being attacked and still no crash.

What i haven't done yet is load your stuff and go to the exact location you say i just haven't had the time to do that yet.

I've even asked my own community and other community's about this and have no other reports yet. But ill keep asking around see what i can find out.

I'm off now the weekend ill try and get a ton of debugging going see if i can find this issue. (depending if i get called into work or not)

[GeryonTom]

Updated with arma log and dump. Summary of the arma log. Note this time I was in windows mode.

"SpawnCheck Local.Agents: 12/15, NearBy.Agents: 2/30, Global.Agents: 0/300, W.holders: 3/80, (radius:106.05m 83fps)." "DAMAGE: player hit by z_hunter in body with zombie for 1.03077 scaled 463.847 Conscious true" "DAMAGE: player hit by z_hunter in body with zombie for 1.34321 scaled 604.444 Conscious true" "DAMAGE: player hit by z_hunter in body with zombie for 0.587065 scaled 264.179 Conscious true" "SpawnCheck Local.Agents: 12/15, NearBy.Agents: 2/30, Global.Agents: 0/300, W.holders: 3/80, (radius:106.05m 91fps)." "DAMAGE: player hit by z_hunter in hands with zombie for 0.957225 scaled 430.751 Conscious true" "DAMAGE: player hit by z_hunter in hands with zombie for 1.24751 scaled 561.381 Conscious true" "SpawnCheck Local.Agents: 12/15, NearBy.Agents: 2/30, Global.Agents: 0/300, W.holders: 3/80, (radius:106.05m 71fps)." "DAMAGE: player hit by z_hunter in legs with zombie for 1.26282 scaled 568.27 Conscious true" "SpawnCheck Local.Agents: 12/15, NearBy.Agents: 2/30, Global.Agents: 0/300, W.holders: 3/80, (radius:106.05m 91fps)." "SpawnCheck Local.Agents: 12/15, NearBy.Agents: 2/30, Global.Agents: 0/300, W.holders: 3/80, (radius:106.05m 71fps)." "Zombie: z_hunter, Distance: 8.36456, Target Reason: Sight-false,99/Sound-true,19" "SpawnCheck Local.Agents: 12/15, NearBy.Agents: 2/30, Global.Agents: 0/300, W.holders: 3/80, (radius:106.05m 83fps)." "SpawnCheck Local.Agents: 12/15, NearBy.Agents: 2/30, Global.Agents: 0/300, W.holders: 3/80, (radius:106.05m 67fps)." "SpawnCheck Local.Agents: 12/15, NearBy.Agents: 2/30, Global.Agents: 0/300, W.holders: 3/80, (radius:106.05m 83fps)." "Zombie: z_hunter, Distance: 4.25643, Target Reason: Sight-false,72/Sound-true,5" "SpawnCheck Local.Agents: 12/15, NearBy.Agents: 2/30, Global.Agents: 0/300, W.holders: 3/80, (radius:106.05m 83fps)."

"DAMAGE: player hit by z_hunter in body with zombie for 0.975296 scaled 438.883 Conscious true"

---

Exception code: C0000005 ACCESS_VIOLATION at 005A70E3 Allocator: C:\Program Files (x86)\Steam\steamapps\common\Arma 2 Operation Arrowhead\dll\tbb4malloc_bi.dll graphics: D3D9, Device: NVIDIA GeForce GTX 980, Driver:nvd3dum.dll 9.18.13.4725 resolution: 1919x1040x32

[GeryonTom]

Much thanks for Skydive's patience on looking at this problem with me. We copied over the walls from US 434 and we were able to reproduce the issue multiple times.

At one point we only had 5 pieces in Stary and we were able to cause the disconnect. We though it might be clipping of fences with the existing Chernarus buildings so we cleared the remaining and proceeded to replicate the clipping. Unfortunately we were not able to reproduce with the new fence.

After we added one of the five fences (one that clipped) and we still could not reproduced. Therefore we suspect it's one or a combination of the remaining four that is causing the disconnects. We'll proceed with the investigation tomorrow.

Cheers,

Geryon

[GeryonTom]

Thank you Luke as well for narrowing this down so that we could focus on the root cause. Several of the players suspected but no one could narrow it until you did last weekend.

You'll have a lifetime supply of NVGs as long as you play on US 434. Really? No not really, only on the test server. :-) Good night.

[Lukethero]

Well thanks Tom, but really what I did was minor in comparison to the work you are doing. Second, who needs your charity NVGs? Found 5 today alone.

[R4Z0R49]

K well shout when you have a root cause i can fix :-)

[GeryonTom]

Okay I'm pretty I know what's causing the crashing. It's the following classnames: MetalFence_base and WoodenFence_base. I noticed that these are not rendered when the server is up. I only had one object and the client would crash. After removing the base class objects even with all the walls present the client remained stable.

I can't comment on how the base objects got into the database (never built them) but I'm pretty certain the fact that these don't render when the server is running has to do with the client crashes.

[R4Z0R49]

As far as i know both those objects are set to private they should not even be allowed to spawn under arma rules.

[GeryonTom]

They were present in the US434 database. As to how they got there I'm not sure. People have been building since they were able and it could be a legacy issue from a previous patch. Whether the base classname gets added in 1.8.3 I'm not sure. For now I'll purge the base classnames from the database and monitor if players building will create more of these.

[R4Z0R49]

i wonder if that base item was named _base in one of the early versions and we changed it later on. I cant think of any other way it would have been in your db without someone hacking

[GeryonTom]

That's what I was thinking but at this moment the past is the past (well at least for me, it's been an on going hair pulling 3 month odyssey). It seems 1.8.3 isn't creating these items so I'm willing to let this go. Especially within a week as players start building again they don't see issues. Hacking was another possibility. I know there's disruptive code out there that can simulate these mass client crashes. It's possible their way of doing this is to adding non-rendering base classes to the object table.

[R4Z0R49]

You can clear them pretty easy within the server.pbo or using an event in mysql.

[Lukethero]

Pops champagne about time we got this figured out. I probably would have killed someone if my client crashed one more time. Long time coming, but glad it is over.

[R4Z0R49]

Now onto the thousand other real bugs :-)

[R4Z0R49]

Ok seems this is a real problem im just not sure right now how *_base items are being created. i've sent a server side fix for publishing the items just waiting on more info ie has it fixed the issue for now.

https://github.com/DayZMod/DayZ/commit/e62c8924df9c12b52a4ea55d8bc0073ddaf5032e

Just to stop _base items being published and remove if built.

Server admins should be able to narrow this down faster by searching your serverlogs looking for any published item with _base in.

[Lukethero]

Just for documents sake, the problem is still active, not a legacy bug. Sure many of you are aware, but just want to get it on pen and paper.

[R4Z0R49]

The issue is open so really no point in that but Ok.

Im on 2weeks of nights so wont be doing anything on this. Admins will have to debug and report what they find or remove base building for now.

[MercianUK]

Re: #356 that has now been closed ,

Suggestion to remove all fence bases: repeat the question before topic closed: Will this detroy all fences?

[kichilron]

As mentioned in #354 it should not make a difference, shouldn't be there in the first place (which is what is causing the issue)

[R4Z0R49]

Early reports of the fix to the server.pbo has fixed the issue for now _base items spawning.

I still haven't been able to recreate how they are spawning to start with still looking into that one.

[KPanicEKS]

Could someone that has access to a spawn menu be injecting them in by chance?

[R4Z0R49]

Really not sure i tested myself on my own clear db created 10 fence's upgrade each one with no _base items being created in the DB.

What i don't understand is how server admins are not noticing private items trying to spawn they spam the server logs on server restart. I've added a new publish block system to block only the _base items while i try to find the root cause. ill also add a system in the server monitor to remove the items if found in the db.

The only other item i can think is maybe the dayz_buildings on server have not updated as it should and maybe still having the _base items as public. Admins would need to open there dayz_buildings and see if fences are set correcting in the configs.

I really cant see whats going just now but ill keep trying to find the issue.

[kichilron]

Most of the admins these days don't bother with logs anyway. They either have an AH that doesn't require looking at logs or they don't even know how to read them anyway.

Or both, which is very likely.

[KPanicEKS]

The only reason I mention the spawn menu is several weeks ago a squad mate found a tent that had "cooked meat" in it, which as we know was replaced with the actual said animal meat, i.e. "cooked beef", "cooked chicken" etc... so I can only assume that the items were spawned in. If the build items are still accessible via some backend system that cheaters are using that could be the source of these erroneous database items. just taking a stab in the dark...

[GeryonTom]

We were able to reproduce the issue on an empty server. It took 4 of us with some admin (Skydive) help to accelerate the accumulation of the building pieces. It definitely wasn't something we could do in 5 minutes. Not saying cheating isn't another possibility but I'm pretty confident that the current issue originates from the mod with legitimate user behavior.

I'll take a look at the logs going forward. Been leveraging Facoptere's parser lately so I've been pretty lazy with the daily log review.

[R4Z0R49]

Witch would mean the objects are being created somehow by the users.

P

[R4Z0R49]

Damit pressed comment.

I need to know if the _base items are still Being created in the db.

[GeryonTom]

Have a few here that might be interesting. I have the full logs if needed.

Here's the link: https://www.dropbox.com/s/h57t68u7plpfelj/Logs_fence.txt?dl=0

[R4Z0R49]

All i really need atm is the amount of _bsse classnames you have in your db and how many publish logs you have in you serverlog. Im at work i cant download links. For the next two weeks im on nights.

[fazmond]

you want to test see if its to do with the removal of a wall that's creating the _base because 1x _base entry reappeared in my database after the removal of a wall piece to put a gate in its place and it was a removal of WoodenFence_Foundation that caused it not a full wall piece

[R4Z0R49]

Ok thanks faz ill look into that i do see something within that script that may be causing the issue.

[R4Z0R49]

Ok thanks to faz i've been over the pulldown script and found the issue

611c2c6c503eef793d5b3b1a423eb11fe4bb50d2

I'm still at work so cannot test this. till the weekend maybe someone else can.

[R4Z0R49]

Soon as i get a confirmation its fixed ill be able to close this.

[R4Z0R49]

Issue resolved.