Server Lag After 1.52 Patch

[LouDnl]

Could be Bohemia causing this, could be a part of code in the mission. Creampie and I have been testing multiple solutions but all without any succes. Disabled addons, disabled mArma, disabled ASM, changed basic.cfg etc. Don't really know where to look now.

ref: http://forums.a3wasteland.com/index.php?topic=2150.0

[LouDnl]

Also created a feedback ticket: http://feedback.arma3.com/view.php?id=26108

[dagg929]

Tried swapping to default mission, no better-- when server is under load and full, it just randomly cuts out for however many seconds. Server FPS is good, steady, I get on our server always 15+ fps until this suddenly occurs.

Upvoting topic on A3 Feedback. Surprised to not see as much noise on the A3 forums about this, seems to be hitting everybody, regardless of mod. (Exile and Wasteland at least.)

[LouDnl]

Yes I have had this on Exile too, but not as much as on A3W. I am getting almost zero response in the server hosting Skype group. I have disabled ASM now to see if that's the cause.

[dagg929]

I'm wondering if the reason I'm seeing this on Exile is because it runs on the same box as my A3Wasteland. Might be occurring at the same time? Talked to a few Exile server admins and they are not getting this.

[LouDnl]

I'm running all my servers on separate boxes. My Exile doesn't have much players but i've seen this choke atleast once. Do you run ASM on your servers?

[dagg929]

Nope! I did turn it on however to try and monitor these issues just yesterday, but we were getting them before. It isn't ASM

[dagg929]

I also stopped running HC as we get it with or without, just to make the problem more simple.

[LouDnl]

Well disabling ASM didn't help. Still getting the drops.

[dagg929]

Did you notice a hike in bandwidth usage just prior to this? Wondering if they just screwed up and when this is hanging, it is actually trying to utilize a ridiculous shitton of down/upload that is causing these hangs.

[LouDnl]

Not really noticed it, but could keep an eye out for it.

[dagg929]

We had this happen I swear a few months back. I had to down my slots to like 56 versus 64 because the server was using a ton of bandwidth whereas it hadn't before. It would hit a certain point, then we'd get hangs like we are getting now-- only now it doesn't matter if it is a small playercount or a large one.

[LouDnl]

Yea I had that too. That's this issue: http://feedback.arma3.com/view.php?id=23593

[AgentRev]

lol BIS denying any wrongdoing and blaming mods, classic

[LouDnl]

Yea as usual...

[hobart666]

Any lawyers in the Community? We should go after them class-action style... Seriously.

[dagg929]

It's at least acknowledged now, huzzah!

[AgentRev]

From past experience, acknowledged = "we don't give a crap, go away"

[LouDnl]

We've been monitoring Wasteland and Exile (GamersInc). As we can see this issue does not occur on Exile. It does however occur on Wasteland. But when the server reaches 55+ (or something) players it doesn't seem to happen. Occurrences seem to be random. I think however it is either tie in to players connecting/disconnecting or to the mission system. But no hard proof there. We (Creampie and I) think that it might be some decrepated function which is still used and is acting up. But which? No frigging idea. It's very annoying.

[LouDnl]

@dagg929 you wrote earlier that you tried default mission. As in Rev's Vanilla A3Wasteland?

[dagg929]

I did, yes. With ExtDB2. We had it reoccur with it around 30 people.

My mission file isn't far off from being default now, though I've been tweaking small things on it such as missions or the like.

I had this occur last night on my box with my server at 63, which then killed it down to 5 people by the time it stopped. (BE timed them all out.) I rebooted it at that point and it came up, was acting funny for a few seconds inititially, then stopped. I'm wondering if this causes a huge net spike and that is what is going on when we see this occur, with it then catching up with itself. Has anyone checked their bandwidth usage during this?

[LouDnl]

Theres a new perf out: https://www.dropbox.com/sh/582opsto4mmr8d8/3BSy9PdRGm

[LouDnl]

Could this be a thing @AgentRev ? https://community.bistudio.com/wiki/BIS_fnc_MP

From version 1.50 the function bis_fnc_mp will use the engine based remote execution. This will positively affect its performance, namely processing speed and amount of transferred data. The syntax and functionality of bis_fnc_mp will remain same to retain the backward compatibility. To prevent abusing of the remoteExec and remoteExecCall commands the bis_fnc_mp will use, the content authors will be able to define the operation modes as well as white-lists for commands/functions for clients and server separately. White-lists defined in CfgRemoteExecCommands and CfgRemoteExecFunctions won't be supported from 1.48 as they will be replaced by the new structure more powerful and detailed structure: CfgRemoteExec.

and

Pay close attention to where and when you call BIS_fnc_MP. If you use it on the server during init, it will probably run before the player is loaded into the server, and the player behaves more like a JIP. Either waiting for the briefing to end (time > 0), or using the JIP parameter in BIS_fnc_MP are two ways of getting around this issue.

It's used in several spots across the mission.

[LouDnl]

Or maybe it's this: https://community.bistudio.com/wiki/BIS_fnc_MPexec

Which is used in filterExecAttempt.sqf ?

[LouDnl]

These files use either BIS_fnc_MP or BIS_fnc_MPexec

initPlayerLocal.sqf > vanilla mission initServer.sqf > vanilla mission 3Dmarkers.sqf > non vanilla addon APOC_cli_startAirdrop.sqf > non vanilla addon va_functions.sqf > non vanilla addon filterExecAttempt.sqf > vanilla mission fn_vehicleInit.sqf > vanilla mission fn_MPexec.sqf > vanilla mission updateConnectingClients.sqf > vanilla mission

Would be nice to know how and to what to replace these functions.

[LouDnl]

More information here: https://community.bistudio.com/wiki/Arma_3_Remote_Execution

[dagg929]

Anyone get this yet with the newest perf branch? Comment from Dwarden on the newest: + more crashfixes/freezefixes

Here's hoping.

[LouDnl]

Yes still getting them.

[LouDnl]

Well I just did a dry test run with a test server. When first missions start there is a choke. Seems like with every mission start theres some sort of choke. When I disabled all missions the server runs without chokes for 1.5hours+.

Something within the mission system MUST be choking the server. @AgentRev any help here is much appreciated.

[LouDnl]

Ofcourse it's always a mix of things. But some chokes definatly come from the mission system. Well atleast it looks like it does. No frigging idea why though.

[LouDnl]

Ok so I disabled all addons I could find in (client/server)/init.sqf including r3f, far revive, playericons etc etc. Disabled territories, disabled server cleanup. Disabled all missions except 1, MBT.

What I see is when the starts up and the mission controllers load theres a micro choke (first red dot) of about a second. Then as soon as the MBT mission starts the choke happens (red line). http://i.imgur.com/lQ1QadV.png

It seems there is something buried in the missioncontroller/missions that chokes the server.

[dagg929]

Is this mission specific? Do you get the same with another mission?

[LouDnl]

Also getting this with other missions yes.

[dagg929]

So the thought here is that, with players, the server is hanging because of a mission call referencing a bad line of code somewhere-- or a depreciated call.

Going through my logs now to see if my freezes match the time in my RPT that it is attempting to spawn a mission.

[dagg929]

Not able to see the relation on my side but I'll document the freezes more closely and be on the look out now for this.

[LouDnl]

When I use ASM to check the freeze time and cross reference that with the rpt log its in 7 of the 10 counts at the exact same time a mission starts. But theres also 3 times its not. So it's really strange

[LouDnl]

The strange thing is, the more players on the server, and thus lower the fps the less frequent the problem.

[AgentRev]

If you can provide the RPT log with the time at which the freeze occurs, I can investigate.

[LouDnl]

Sure i'll get you one. But there's nothing much to see though.

[LouDnl]

Here's a log. Theres nothing to see there though :( I disabled all addons. The choke happened right at the mission start. Only 1 mission in the array. https://drive.google.com/file/d/0By5PInQAIAA6QVNPYm1MY3FLeW8/view?usp=sharing

[AgentRev]

What's the timestamp of the freeze?

[LouDnl]

Right on or right before 21:12:54. You can see the mission loading from disk in ASM and then it chokes. And after about 10 seconds it comes back.

[LouDnl]

Weird thing is though, with less players on it seems to happen more often then with more players on. I've even seen chokes of 30 minutes.

[AgentRev]

21:12:54

RPT starts at 22:00:42

[LouDnl]

wait huh

[LouDnl]

Fuck me wrong link: https://drive.google.com/file/d/0By5PInQAIAA6ZXNEOUxGa3IxRXc/view?usp=sharing

[LouDnl]

That log I linked before was for Dwarden. Had chokes too, but don't know the timestamps on that one.

[LouDnl]

Ill upload a log of the running server too. Got some timestamps on that one also

[LouDnl]

https://drive.google.com/file/d/0By5PInQAIAA6Tk5feHpsNEVjcnM/view?usp=sharing Timestamps: 18:50:52 19:31:15 19:32:52 20:03:37

[LouDnl]

Theres more timestamps, but cant get them all

[AgentRev]

Well, can't reproduce on my own comp, no chokes at all...