Possible performance hit on city activation

GoldJohnKing commented 4 years ago

Arma 3 Version: 1.98 (stable) CBA Version: 3.15.1 (stable) ACE3 Version: 3.13.3 (stable) RHS USAF Version: 0.5.2 (stable) Hearts and minds Version: master_daily (0f9273c)

Mods:

- CBA_A3
- ace
- RHS USAF
- RHS AFRF

Description:

I've found some possible performance hits on city activation, which are ace_interact_menu_fnc_addActionToClass, btc_fnc_mil_create_group and btc_fnc_ied_create.
These functions are executed in one frame, causing both client and server freeze when city activates.
I wonder if these functions could be done in different frames to get a better performance.

Steps to reproduce:

Teleport or move to first time activate a city.
You would encounter performance hit on both server side and client side, and you can see other players warp around during the activation process. If the server is not that high-end, server and client would freeze for a few seconds.
The performance of second time activation of a city could be better, but still cause stuttering.

Add mission file:

I use master_daily branch with RHS AFRF Russian (VDV) as enemy.

Screenshots: 屏幕截图(82)

屏幕截图(2)

屏幕截图(3)

Vdauphin commented 4 years ago

Missing mission file and RPT in addition you could also drop the tracy log file

GoldJohnKing commented 4 years ago

Tracy and mission files here, the performance hit should be at frame 7154.

RPT file with debug=true is not currently avaliable, I would upload it once I have one. And my server use -noLogs parameters, so it might not affected by logging performance.

Related Files.zip

Vdauphin commented 4 years ago

Never disable log. Those does not affect performance at all Please upload a rpt, mission file and tracy when the issue occur

Vdauphin commented 4 years ago

Also you send the wrong mission file and didn't gave all mod you are using So:

If you have found an issue with Hearts and Minds please make sure that Hearts and Minds really is the cause of the problem. To do this try to reproduce the issue with using only @CBA_A3, @ace and @RHSUSAF on a stable and unchanged Hearts and minds available here on a dedicated server.

And 94ms for a city activation is pretty fast... When you know that is cached for the rest of the game (I mean, until you reset the database so even after a server restart), it is fine. I don't know if you activated the same city but the second call of city/activate.sqf take 5ms...

server and client would freeze for a few seconds.

I am still surprise because there are no frame duration upper than 100ms during the city activation

GoldJohnKing commented 4 years ago

Also you send the wrong mission file and didn't gave all mod you are using

Yes, my server use a different mission file and mod list. I run the test locally using the uploaded mission file and not that much mods to keep away from the effects caused by mods, and the stutter keeps showing up. So I uploaded the local testing mission file.

And 94ms for a city activation is pretty fast... When you know that is cached for the rest of the game (I mean, until you reset the database so even after a server restart), it is fine. I don't know if you activated the same city but the second call of city/activate.sqf take 5ms...

Yes, I get that. The first time ever of city activation would cause discontinuous freezes in a few seconds, and second time city activation would be much better, only stuttering happens.

Please upload a rpt, mission file and tracy when the issue occur

Ok, I'll get a new one corresponding to each other.

GoldJohnKing commented 4 years ago

I found when a city first activates, there are huge amounts of guaranteed packets (shown as G when #monitor 1 is on) transfered between server and clients, sometimes the amount would goes up to 10^6. The stuttering happens when the guaranteed packets are above 0, and seems to be more severe when the guaranteed packets become more. Could it be related to the stuttering?

GoldJohnKing commented 4 years ago

RPT, mission file and tracy.zip

Vdauphin commented 4 years ago

Yes, my server use a different mission file and mod list. I run the test locally using the uploaded mission file and not that much mods to keep away from the effects caused by mods, and the stutter keeps showing up. So I uploaded the local testing mission file.

If mod are the cause, you should disable them. Use tracy on you mission and you will see witch mod freeze your game.

Yes, I get that. The first time ever of city activation would cause discontinuous freezes in a few seconds,

I don't notice a freeze during the city activation in your tracy but a freeze after. Can you in "trace parameters" enable "instructionprofilling"?

I found when a city first activates, there are huge amounts of guaranteed packets (shown as G when #monitor 1 is on) transfered between server and clients, sometimes the amount would goes up to 10^6. The stuttering happens when the guaranteed packets are above 0, and seems to be more severe when the guaranteed packets become more. Could it be related to the stuttering?

That is really strange. Is it from the mission you send me?

GoldJohnKing commented 4 years ago

If mod are the cause, you should disable them. Use tracy on you mission and you will see witch mod freeze your game.

I'm using only RHS*4 + CBA + ACE + Tracy required mods and ArmaZeusCache to run the test, and the stuttering still happens, so it seems not caused by mods.

Can you in "trace parameters" enable "instructionprofilling"?

I did not found that parameter in tracy, where do I set that?

That is really strange. Is it from the mission you send me?

Yes, and it do not always happens. It seems that object creation (ied, enemy, civ, vehicle) would generate guaranteed packets. Does it related to basic.cfg of the server? My basic.cfg follows default value.

Vdauphin commented 4 years ago

I did not found that parameter in tracy, where do I set that?

Sans titre

Does it related to basic.cfg of the server?

No idea, never have this issue

GoldJohnKing commented 4 years ago

File too big, so uploaded to mega.

https://mega.nz/file/XiRnVQqJ

I also use debug graph in ace self-interaction menu and see once the delay reaches zero (city activation and unit creation finished), the stutter stops immediately.

Vdauphin commented 4 years ago

mega is asking me a decryption key x)

GoldJohnKing commented 4 years ago

What about this one?

https://mega.nz/file/XiRnVQqJ#4c1Ef61Yi2SfrOAVbFlJrHnqvn5qYdadLLn-7lXqYUI

Vdauphin commented 4 years ago

perfect!

GoldJohnKing commented 4 years ago

I've tested for several days, #922 greatly reduces stuttering/freezing issue on city activation!

I've done some comparation and found the most noticeable stuttering/freezing are caused by the first time ever activation, reactivation of a formal activated city does not impact fps that much.

I wonder if a preload feature is much more capable for solving the first time activation stuttering issue. That is to say, activate cities at first time ever of server start and save the city in database, this might increase first time server start time, but could greatly reduce stuttering in game.

I see different types of city carries different amount of enemies and civilians. Bigger the city, more units to be spawned, which means more performance hit. The preload feature could take advantages of city types, that is to preload only cities carrying large amount of units, such as Capital and Airport, but leave Village and Local city load in game. The amount of units to be spawned could also used for judging if a city should be preloaded. It would be better if server admins could choose to enable/disable and set preload city types in mission parameters.

Vdauphin commented 4 years ago

I've done some comparation and found the most noticeable stuttering/freezing are caused by the first time ever activation, reactivation of a formal activated city does not impact fps that much.

Honestly the freeze is very tiny server side, always around 100ms. There is a freeze player time longer but is simply due to the fact that objects (like units and vehicles) are not loaded in the ram so there is hard drive access.

I wonder if a preload feature is much more capable for solving the first time activation stuttering issue. That is to say, activate cities at first time ever of server start and save the city in database, this might increase first time server start time, but could greatly reduce stuttering in game.

I thought about that but H&M isn't design like that. And I don't think this will resolve the issue. I still think the freeze is due to Arma not fully load texture, objects in ram so you have a delay player side.

Bigger the city, more units to be spawned, which means more performance hit.

Not necessary. Now unit creation is delayed carefully so there are not more unit created in the same time. For me server side there are no freeze. There might be a freeze player side but it more due to how Arma manage preloading stuff.

The preload feature could take advantages of city types, that is to preload only cities carrying large amount of units, such as Capital and Airport, but leave Village and Local city load in game. The amount of units to be spawned could also used for judging if a city should be preloaded. It would be better if server admins could choose to enable/disable and set preload city types in mission parameters.

So design all the H&M to support such feature is a little overhead to me.

GoldJohnKing commented 4 years ago

Okay, I get your point. Since #922 was merged, I'll close the issue.

Vdauphin commented 2 years ago

For reference https://github.com/Vdauphin/HeartsAndMinds/compare/master...Vdauphin:FIX-client_shuttering?expand=1 Deployed on my dev server, need to be tested during 1.22 debugging

Vdauphin / HeartsAndMinds

Possible performance hit on city activation #921