Headless Client testing results

[invrecon]

This is a long read, but bear with me. I'm not asking for a feature, just reporting my experience from some testing I did and looking for your thoughts.

We're a small group (3-5 players usually) and play on a dedicated server. I like to grab the latest source and compile the missions, so we're pretty much on the bleeding edge. I noticed that recently the performance seems to have worsened a bit. It usually manifests in the AI moving in a laggy way and dying with quite a bit of delay after being shot, and actions like transferring items between inventories taking longer. Naturally, all of this was caused by the server FPS dropping badly during these moments (to about 1-2 for a couple of seconds, then going up to a measly 10-15 before stabilizing at around 30). It seems especially bad on the Livonia terrain, but is generally observable everywhere.

So I went looking through the code for anything that might cause it. My first guess was a bad while loop somewhere that doesn't sleep between iterations, but I didn't find a single one, the code is very solid. The second option I looked into was AI. To verify that theory, I commented out the initialization of ambient infantry, traffic and so on. This resulted in excellent performance with server FPS at 50. So it looks like there's just too much infantry active at some points in the mission and whenever a group spawns or is "triggered" by an engagement, the server gets overloaded. The main issue is that all the AI computation in Arma seems to be done in a single thread. That's also something I can observe on my server, because a single core is working at a 100% while the remaining ones are barely doing anything in comparison. At that point I thought I had hit a dead end, because we can't lower the number of simultaneously active AI without affecting the gameplay, all of their operations running in a single thread is an engine limitation, and I didn't see how this could be easily offloaded to a HC, because AI spawning is not handled by a single script, but several.

After a while I figured that we wouldn't need to offload all AI, just the most impactful. So I tried to add a HC unit to the mission and remotely execute the DRN fn_AmbientInfantry.sqf only on that. To my surprise, it worked and the infantry still spawned, though performance still wasn't where I wanted it. After doing the same for fn_MilitaryTraffic.sqf, which is responsible for both civilian and military traffic, it was much better. Looks like driving is really hard for the AI (surprise) and therefore the server too.

I experimented only with the built Livonia mission (from Build/missionfiles) and the only changes I made were to add a HC unit to mission.sqm (with the editor) and the following:

diff --git a/functions/DRN/fn_AmbientInfantry.sqf b/functions/DRN/fn_AmbientInfantry.sqf
index 6305454..c52091d 100644
--- a/functions/DRN/fn_AmbientInfantry.sqf
+++ b/functions/DRN/fn_AmbientInfantry.sqf
@@ -15,17 +15,19 @@
  *   [_garbageCollectDistance]: Dead units at this distance from _referenceGroup will be deleted.
  *   [_fnc_OnSpawnUnit]: Code run once for every spawned unit (after the whole group is created). The unit can be accessed through "_this". Default value is {}.
  *   [_fnc_OnSpawnGroup]: Code run once for every spawned group (after the whole group is created). The group can be accessed through "_this". Default value is {};
+ *   [_enemyFrequency]: Only required for calling UnitClasses.sqf.
  *   [A3E_Debug]: true if debugmessages and areas will be shown for player. Default false.
  * Dependencies: CommonLib v1.01
  */

-if (!isServer) exitWith {};
+if (isServer || hasInterface) exitWith {};

 private ["_referenceGroup", "_side", "_groupsCount", "_minSpawnDistance", "_maxSpawnDistance", "_infantryClasses", "_minSkill", "_maxSkill", "_garbageCollectDistance"];
 private ["_activeGroups", "_activeUnits", "_spawnPos", "_group", "_possibleInfantryTypes", "_infantryType", "_minDistance", "_skill", "_vehicleVarName", "_factionsArray"];
 private ["_minUnitsInGroup", "_maxUnitsInGroup", "_i", "_atScriptStartUp", "_currentEntityNo", "_DebugMsg", "_farAwayUnits", "_farAwayUnitsCount", "_unitsToDeleteCount", "_groupsToDeleteCount"];
 private ["_DebugMarkers", "_DebugMarkerNo", "_DebugMarkerName", "_isFaction", "_unitsToDelete", "_groupsToDelete", "_tempGroups", "_tempGroupsCount"];
 private ["_fnc_OnSpawnUnit", "_fnc_OnSpawnGroup"];
+private ["_enemyFrequency"];

 _referenceGroup = _this select 0;
 _side = _this select 1;
@@ -40,7 +42,10 @@ if (count _this > 9) then {_maxSkill = _this select 9;} else {_maxSkill = 0.6;};
 if (count _this > 10) then {_garbageCollectDistance = _this select 10;} else {_garbageCollectDistance = 750;};
 if (count _this > 11) then {_fnc_OnSpawnUnit = _this select 11;} else {_fnc_OnSpawnUnit = {};};
 if (count _this > 12) then {_fnc_OnSpawnGroup = _this select 12;} else {_fnc_OnSpawnGroup = {};};
+if (count _this > 13) then {_enemyFrequency = _this select 13;} else {_enemyFrequency = 2;};

+// Need to do this again, because we need its global variables and it is only executed on the server
+[_enemyFrequency] call compile preprocessFileLineNumbers "Units\UnitClasses.sqf";

 //WHY!?!?!?!?!
 _factionsArray = [A3E_VAR_Side_Ind , A3E_VAR_Side_Ind , A3E_VAR_Side_Ind , A3E_VAR_Side_Ind , A3E_VAR_Side_Ind , A3E_VAR_Side_Ind , A3E_VAR_Side_Opfor , A3E_VAR_Side_Opfor , A3E_VAR_Side_Opfor , A3E_VAR_Side_Opfor ,A3E_VAR_Side_Opfor];
diff --git a/functions/DRN/fn_MilitaryTraffic.sqf b/functions/DRN/fn_MilitaryTraffic.sqf
index 5f9b1bd..2c477b7 100644
--- a/functions/DRN/fn_MilitaryTraffic.sqf
+++ b/functions/DRN/fn_MilitaryTraffic.sqf
@@ -1,3 +1,5 @@
+if (isServer || hasInterface) exitWith {};
+
 private ["_activeVehiclesAndGroup", "_vehiclesGroup", "_spawnSegment", "_vehicle", "_group", "_result", "_possibleVehicles", "_vehicleType", "_vehiclesCrew", "_skill", "_minDistance", "_tries", "_trafficLocation"];
 private ["_currentEntityNo", "_vehicleVarName", "_tempVehiclesAndGroup", "_deletedVehiclesCount", "_firstIteration", "_roadSegments", "_destinationSegment", "_destinationPos", "_direction"];
 private ["_roadSegmentDirection", "_testDirection", "_facingAway", "_posX", "_posY", "_pos"];
diff --git a/functions/Server/fn_initServer.sqf b/functions/Server/fn_initServer.sqf
index d3066ac..f24111a 100644
--- a/functions/Server/fn_initServer.sqf
+++ b/functions/Server/fn_initServer.sqf
@@ -327,7 +327,7 @@ private _UseMotorPools = Param_MotorPools;
    _radius = (_enemySpawnDistance + 500) / 1000;
    _infantryGroupsCount = round (_groupsPerSqkm * _radius * _radius * 3.141592);

-   [_playerGroup, A3E_VAR_Side_Opfor, a3e_arr_Escape_InfantryTypes, _infantryGroupsCount, _enemySpawnDistance + 200, _enemySpawnDistance + 500, _minEnemiesPerGroup, _maxEnemiesPerGroup, _enemyMinSkill, _enemyMaxSkill, 750, _fnc_OnSpawnAmbientInfantryUnit, _fnc_OnSpawnAmbientInfantryGroup, A3E_Debug] spawn drn_fnc_AmbientInfantry;
+   [_playerGroup, A3E_VAR_Side_Opfor, a3e_arr_Escape_InfantryTypes, _infantryGroupsCount, _enemySpawnDistance + 200, _enemySpawnDistance + 500, _minEnemiesPerGroup, _maxEnemiesPerGroup, _enemyMinSkill, _enemyMaxSkill, 750, _fnc_OnSpawnAmbientInfantryUnit, _fnc_OnSpawnAmbientInfantryGroup, _enemyFrequency, A3E_Debug] remoteExec ["drn_fnc_AmbientInfantry"];

     // Initialize the Escape military and civilian traffic
@@ -417,7 +417,7 @@ private _UseMotorPools = Param_MotorPools;
        };
    };

-   [civilian, [], _vehiclesCount, _enemySpawnDistance, _radius, 0.5, 0.5, _fnc_onSpawnCivilian, A3E_Debug] spawn drn_fnc_MilitaryTraffic;
+   [civilian, [], _vehiclesCount, _enemySpawnDistance, _radius, 0.5, 0.5, _fnc_onSpawnCivilian, A3E_Debug] remoteExec ["drn_fnc_MilitaryTraffic"];

    // Enemy military traffic
@@ -443,8 +443,8 @@ private _UseMotorPools = Param_MotorPools;
    [_vehiclesCount,_enemySpawnDistance,_radius,_enemyMinSkill, _enemyMaxSkill] spawn {
        params["_vehiclesCount","_enemySpawnDistance","_radius","_enemyMinSkill", "_enemyMaxSkill"];
        sleep 60*15; //Wait 15 Minutes before heavy vehicles may arrive 
-       [A3E_VAR_Side_Opfor, [], _vehiclesCount/2, _enemySpawnDistance, _radius, _enemyMinSkill, _enemyMaxSkill, drn_fnc_Escape_TrafficSearch, A3E_Debug] spawn drn_fnc_MilitaryTraffic;
-       [A3E_VAR_Side_Ind, [], _vehiclesCount/2, _enemySpawnDistance, _radius, _enemyMinSkill, _enemyMaxSkill, drn_fnc_Escape_TrafficSearch, A3E_Debug] spawn drn_fnc_MilitaryTraffic;
+       [A3E_VAR_Side_Opfor, [], _vehiclesCount/2, _enemySpawnDistance, _radius, _enemyMinSkill, _enemyMaxSkill, drn_fnc_Escape_TrafficSearch, A3E_Debug] remoteExec ["drn_fnc_MilitaryTraffic"];
+       [A3E_VAR_Side_Ind, [], _vehiclesCount/2, _enemySpawnDistance, _radius, _enemyMinSkill, _enemyMaxSkill, drn_fnc_Escape_TrafficSearch, A3E_Debug] remoteExec ["drn_fnc_MilitaryTraffic"];
     };

    private ["_areaPerRoadBlock", "_maxEnemySpawnDistanceKm", "_roadBlockCount"];

So basically in the server init file replacing the spawn for the mentioned functions with remoteExec on all machines, and in the files themselves making sure they terminate if they run anywhere other than the HC. That was the easiest way I found to make them run on the HC. I also had to pass in another variable to run a script again that is otherwise only executed on the server and generates a couple of global variables that the ambient infantry & military traffic script also need, which I therefore also have to execute on the HC. That's basically it, and it runs with almost no additional errors. Just something in the function call that equips the spawned soldiers, but with a bit of effort I could probably figure out which global variable (or similar) is missing on the HC and causes this.

I noticed no adverse effects on gameplay so far, and performance is simply fantastic. The server now runs consistently at 48-50 FPS, with a few drops down to 30 in some situations, but never below that. But since most of the AI now runs on the HC, server FPS aren't relevant to their performance anymore, now it's how busy the HC is that determines this. And it's looking really good. Instead of seeing a single core scrambling to keep up, the load is much more balanced now.

It's still possible to overload it slightly and make the AI lag for an instant when driving at speed, but only in very few instances. And that makes sense, because as long as whatever core currently does the computation is not overloaded, we get our 50 server/HC FPS.

Now for the disclaimer: this needs much more testing to make sure it doesn't introduce any other issues. I don't have any easy to reproduce benchmarks for the improvements, it's all based on my observations. My main question here is if you think something like this could work in the mission. The difficulties I see are:

1. Adding one (or several) HC entities to the mission.sqm, for every single terrain (unless you have a better way of making changes to this, I haven't looked into it)
2. Some additional scripting to make sure the mission works with an HC connected and without
3. In the case of adding several HC, distributing the load over them. Something relatively simple would be to say "put ambient infantry on one, and military traffic on the other". But that's not a priority, even one HC makes a dramatic difference.

[CaptainPStar]

this needs much more testing to make sure it doesn't introduce any other issues. That is the main reason I didn't added HC myself yet (albeit I experimented with it a bit).

I am also a bit worried about the global AI commanding script (search leader) sending commands to some units. it is all locally now and I am afraid it may result in patrols in HC becoming unresponsive to commands (for example fleeing if arti is incoming).

[invrecon]

Assuming that commands like allGroups and a3e_fnc_OrderSearch work correctly on the "full" picture of the shared mission state across the server and all clients (which I'm really hoping they do), I'd be slightly optimistic that the commanding script still works. But I have much less experience with this than you do.

That's exactly why I'm asking, because I just don't know the mission well enough to judge this. But if you already experimented and aren't confident enough that it could work, there's little chance that I manage to do it. I'm definitely going to test it for a bit and will report back if there's anything groundbreaking, but I don't think that's going to be the case. I'm closing this, feel free to reopen if you want to discuss more. Thanks!

[invrecon]

I've got some more results and they're pretty surprising. First of all, I have to come back to something I said previously:

though performance still wasn't where I wanted it. After doing the same for fn_MilitaryTraffic.sqf, which is responsible for both civilian and military traffic, it was much better.

Now I know why "offloading" the military traffic improved performance by so much. It's because it resulted in no vehicles being spawned at all. It looks like you can only spawn infantry units on HC, although I haven't been able to confirm that, as the only question I found on that topic hasn't been answered. But it would explain why the ambient infantry appears to work and the traffic doesn't.

This left me scratching my head, because to get a noticeable improvement, we really have to do something about the military traffic. I stumbled upon setGroupOwner, learning that this is all it takes to move all units of an AI group to a given client. This is in many ways a much better solution than using remoteExec to run the whole infantry/vehicle spawning script on the HC. First of all, we can use it with less changes to the code:

diff --git a/functions/DRN/fn_AmbientInfantry.sqf b/functions/DRN/fn_AmbientInfantry.sqf
index 6305454..c36007f 100644
--- a/functions/DRN/fn_AmbientInfantry.sqf
+++ b/functions/DRN/fn_AmbientInfantry.sqf
@@ -26,6 +26,10 @@ private ["_activeGroups", "_activeUnits", "_spawnPos", "_group", "_possibleInfan
 private ["_minUnitsInGroup", "_maxUnitsInGroup", "_i", "_atScriptStartUp", "_currentEntityNo", "_DebugMsg", "_farAwayUnits", "_farAwayUnitsCount", "_unitsToDeleteCount", "_groupsToDeleteCount"];
 private ["_DebugMarkers", "_DebugMarkerNo", "_DebugMarkerName", "_isFaction", "_unitsToDelete", "_groupsToDelete", "_tempGroups", "_tempGroupsCount"];
 private ["_fnc_OnSpawnUnit", "_fnc_OnSpawnGroup"];
+private ["_hc", "_localityChanged"];
+
+_hc = owner hc1;
+diag_log formatText ["### Acquired client ID of HC in ambient infantry: %1", _hc];

 _referenceGroup = _this select 0;
 _side = _this select 1;
@@ -128,6 +132,9 @@ while {true} do {
             //_infantryType createUnit [_spawnPos, _group,"", _skill, "PRIVATE"];
            _group createUnit [_infantryType, _spawnPos, [], 0, "FORM"];
         };
+        
+        //_localityChanged = _group setGroupOwner _hc;
+        //diag_log formatText ["### Moved an infantry group to HC: %1", _localityChanged];

         {
             //setskills
diff --git a/functions/DRN/fn_MilitaryTraffic.sqf b/functions/DRN/fn_MilitaryTraffic.sqf
index 5f9b1bd..724f08f 100644
--- a/functions/DRN/fn_MilitaryTraffic.sqf
+++ b/functions/DRN/fn_MilitaryTraffic.sqf
@@ -3,10 +3,14 @@ private ["_currentEntityNo", "_vehicleVarName", "_tempVehiclesAndGroup", "_delet
 private ["_roadSegmentDirection", "_testDirection", "_facingAway", "_posX", "_posY", "_pos"];
 private ["_fnc_OnSpawnVehicle", "_fnc_FindSpawnSegment"];
 private ["_debugMarkerName", "_allRoadSegments"];
+private ["_hc", "_localityChanged"];
 //private ["_debugMarkers", "_debugMarkerNo", "_debugMarker", "_debugMarkerName"];

 params["_side","_vehicleClasses",["_vehicleCount",10],["_minSpawnDistance",1000],["_maxSpawnDistance",1500],["_minSkill",0.5],["_maxskill",0.6],["_fnc_OnSpawnVehicle",{}],["_debug",false]];

+_hc = owner hc1;
+diag_log formatText ["### Acquired client ID of HC in military traffic: %1", _hc];
+

 //trying around to get traffic working on smaller islands
 //ignoring distance handed over by fn_initServer
@@ -313,6 +317,9 @@ while {true} do {
             _vehiclesCrew = _result select 1;
             _vehiclesGroup = _result select 2;

+            _localityChanged = _vehiclesGroup setGroupOwner _hc;
+            diag_log formatText["### Moved a vehicle group to HC: %1", _localityChanged];
+            
             // Name vehicle
             sleep random 0.05;
             if (isNil "drn_MilitaryTraffic_CurrentEntityNo") then {

We just have to acquire the HC's client ID and can then move the groups as soon as they're created. An additional advantage is that it's less "invasive". The whole script is still executed on the server, just the game's AI logic runs on the HC. This makes it much less likely to accidentally break something. The previous solution required quite a few workarounds for public variables and other stuff that exists on the server but not the HC and is required by the scripts when we execute them remotely. No more of that now.

But here's the downside: while we can now easily (and relatively safely, as far as I can tell) offload the AI, all of the scripted logic still runs on the server. That's fine, as long as we're working under the assumption that most of the load is generated by the AI and not the scripts managing it. But the additional testing I did indicates that this is not actually the case. In the modified mission that moves ambient infantry and military traffic groups to the HC, I still regularly got the dreaded less than 10 server FPS regularly. And it always coincided with a vehicle being spawned, making me suspect that fn_MilitaryTraffic.sqf is the culprit. And it's a likely candidate, after all it does quite a bit of work looking for a suitable spawn position. To confirm my hypothesis, I checked out the vanilla mission again and only modified it not to spawn any military traffic. No HC involved. And while driving around the island I barely ever ran into a situation with less than 40 server FPS, most of the time being at around 48. So I think I was wrong when I thought there was simply too much AI and that we need to offload it. The AI is pretty close to impacting performance (the single core that is most busy during the mission hovers at around 90% load on my server), but it's below the threshold at the moment. For now, I should probably investigate the military traffic script a bit more. Some profiling of where it spends the most time would be great, but unfortunately that's much less convenient to do in Arma than I'm used to from other projects.

[invrecon]

I've done some measurements to figure out where the military traffic script spends all its time. It looks like when the player is more or less stationary, finding a spawn position is most expensive relative to the other stuff. That makes sense, since finding a spawn position when there are already many vehicles spawned around the player takes more attempts. When the player is moving quickly and fewer vehicles are already present around them, most of the time is taken up by the part getting all nearby road segments to determine the first destination with nearRoads. Unfortunately I don't see how this can be substantially improved.

[dookiejones]

I've done some measurements to figure out where the military traffic script spends all its time. It looks like when the player is more or less stationary, finding a spawn position is most expensive relative to the other stuff. That makes sense, since finding a spawn position when there are already many vehicles spawned around the player takes more attempts. When the player is moving quickly and fewer vehicles are already present around them, most of the time is taken up by the part getting all nearby road segments to determine the first destination with nearRoads. Unfortunately I don't see how this can be substantially improved.

Could the performance of this be improved by using roadsConnectedTo? nearRoads used to find closest segment to _vehicle with roadsConnectedTo providing the array?

[invrecon]

That's a smart idea and would probably work in an ideal world, but the first result I got when looking up how that works isn't really encouraging. We could probably get away with some roads not being detected as long as there are enough others, but it might still change how the vehicle spawning works in unpredictable ways, which is a hard sell.

[dookiejones]

Lines 89-93: _roadSegment = _vehicle nearroads 20; _roadSegment = selectRandom _roadSegment; _roadSegments = roadsConnectedTo _roadsegment; _destinationSegment = selectRandom _roadSegments; _destinationPos = getPos _destinationSegment;

This seems to be working for me. I am unable to determine the impact it has on the server. I am an auto mechanic and that is a bit above my skill level.

[invrecon]

This is great, thanks for the suggestion! To measure the difference between the two methods, I used the BIS_fnc_codePerformance function. I went into the editor, loaded the Livonia terrain and placed a player unit inside a vehicle on a road somewhere.

To test the current method of getting road segments, I then ran

["_roadSegments = player nearRoads 2000;   
_destinationSegment = selectRandom _roadSegments;   
_destinationPos = getPos _destinationSegment;"] call BIS_fnc_codePerformance;

in the debug console. Same for your suggestion, which is

["_roadSegment = player nearRoads 20;  
_roadSegment = selectRandom _roadSegment;  
_roadSegments = roadsConnectedTo _roadSegment;  
_destinationSegment = selectRandom _roadSegments;  
_destinationPos = getPos _destinationSegment;"] call BIS_fnc_codePerformance;

Now the first one took an average of 3.5 ms, whereas the second one was only 0.009 ms. That's a factor of almost 390! Because it's getting late I couldn't check whether it achieves more or less the same thing as the old one yet, but performance-wise it's looking very promising.

As for the actual mission performance, I can't give a clear answer either at this point. The problem is that just replacing the instance on line 89 is not enough, because nearRoads is used in the same way on lines 140 and 233+. And these have probably even more of an impact, since they are within loops and can be executed repeatedly for a number of tries until it gives up.

One issue I have for further testing is that the other occurrences aren't as easy to replace as the first one. For the one on line 89, we get a vehicle (which is most likely on a road), so we're pretty much guaranteed to get a road segment with the nearRoads 20 call. That's not the case for the others, because they are relative to the player position (I think), which could be very far away from any roads, resulting in us not finding any. It's a solvable problem, but we'll cross that bridge when we get to it. Before spending any significant amount of time on that though, ideally I would first replace all these instances with dummy code that just cheaply selects random road segments. If there are no more performance issues after that, we can be sure that the nearRoads usages are actually what makes the military traffic script slow and that it's worth to continue looking into it. But the problem is that this will lead to the garbage collection aggressively deleting the vehicles, because they will most likely spawn far away from the player since that method doesn't take the player position into account when spawning them. This constant deleting and spawning will affect the results, rendering them useless. Disabling the garbage collection is not an option, because I need vehicles to be collected from time to time so new ones are spawned and I can observe the performance of that. So it's a bit of a chicken and egg problem: I don't want to spend time making the spawning work properly before being sure that it's worth it, but to test it I need spawning that already works.

I might be able to work around that with completely random spawning of vehicles and replacing the garbage collection with something that just deletes a vehicle every 20 seconds or so, which should cause a new one to be spawned.

[dookiejones]

BIS_fnc_nearestRoad should be able to replace nearRoads when looking for a road near the player. Theoretically it should be faster for everything since it returns Return Value: Object - nearest road object, objNull if not found nearRoads seems to start at the SW corner and work towards NE corner while BIS_fnc_nearestRoad stops at the first road object encountered.

[dookiejones]

@invrecon On my machine, desktop not server, I am getting 3.86873ms for the nearRoads code snippet above. For the BIS_fnc_nearestRoad snippet I get 0.008ms. I get 0.0096ms using ["_roadSegment = [getPosATL _vehicle, 2] call BIS_fnc_nearestRoad; _roadSegments = roadsConnectedTo _roadsegment; _destinationSegment = selectRandom _roadSegments; _destinationPos = getPos _destinationSegment;"] call BIS_fnc_codePerformance;

This could replace most usage of nearRoads as ["_roadSegment = [getPosATL _vehicle, 2000] call BIS_fnc_nearestRoad; _roadSegments = roadsConnectedTo _roadsegment; _destinationSegment = selectRandom _roadSegments; _destinationPos = getPos _destinationSegment;"] call BIS_fnc_codePerformance; Runs in 1.56006ms.

[dookiejones]

Back to the initial topic. On my server I used Werthles headless client script, https://forums.bohemia.net/forums/topic/184485-werthles-headless-module/ for an easy drop in solution. It seems moving the AI to HC after they spawn works in most if not all cases as long as they have their orders re-issued. Werthles handles this by creating a dummy waypoint that is re-issued after locality change. The only thing I have noticed is the rotation of a unit can sometimes change but this is a known limitation of locality change in Arma 3. I have only done maybe 4 hours of testing Werthles on completely vanilla Escape but it seems to handle it beautifully. I use TCL for the AI and that is handled just fine as well.

[dookiejones]

fn_MilitaryTraffic.zip The attached fn_MilitaryTraffic.sqf should remove all instances of nearRoads It appears to be working as far as I can tell. Not sure on the overall impact of the changes, I do not know how to test the whole sqf it just errors if I paste it in debug as above. I made some assumptions here, again I am way above my skill level.

[CaptainPStar]

Keep in mind that _roadSegment = [getPosATL _vehicle, 2000] call BIS_fnc_nearestRoad; _roadSegments = roadsConnectedTo _roadsegment; _destinationSegment = selectRandom _roadSegments; is a completly different logic as it only selects one road near the _vehicle pos and adjacent ones (in arma3 only two), while the old scripts selects a random road somewhere in 2km. Same for the loop where the range is Max range - Min range. For same effect you would have to call the function on a random position (and check if there is an actual road).

[invrecon]

Back to the initial topic. On my server I used Werthles headless client script

I suppose the nice thing about it is that it balances across any number of HCs, which is not the case with what I tried where it's basically hardcoded to one instance. But doesn't it require server, clients and HCs to run the addon?

Keep in mind that ... is a completly different logic as it only selects one road near the _vehicle pos and adjacent ones (in arma3 only two), while the old scripts selects a random road somewhere in 2km.

Unfortunately that's right. I confirmed it with

// Mark road segments found with nearRoads
_segments = player nearRoads 2000;
_i = 0;
{
    _marker = format ["marker%1", _i];
    createMarker [_marker, position _x];
    _marker setMarkerShape "RECTANGLE";
    _marker setMarkerColor "ColorRed";
    _marker setMarkerSize [10, 10];
    _i = _i + 1;
} forEach _segments;

which gives us the nice 2 km circle we expect while

// Mark road segments found with BIS_fnc_nearestRoad & roadsConnectedTo
_roadSegment = [position player, 2000] call BIS_fnc_nearestRoad;
_marker = "firstSegment";
createMarker [_marker, position _roadSegment];
_marker setMarkerShape "RECTANGLE";
_marker setMarkerColor "ColorGreen";
_marker setMarkerSize [10, 10];
_segments = roadsConnectedTo _roadSegment;
_i = 0;
{
    _marker = format ["marker%1", _i];
    createMarker [_marker, position _x];
    _marker setMarkerShape "RECTANGLE";
    _marker setMarkerColor "ColorRed";
    _marker setMarkerSize [10, 10];
    _i = _i + 1;
} forEach _segments;

gives us only the nearest segment (green) and the two adjacent ones as you said.

BIS_fnc_nearestRoad should be able to replace nearRoads when looking for a road near the player.

Great stuff! While I don't think we need to be able to find the nearest road to a player or existing vehicle (usually we want a random one some distance away, both for spawning vehicles out of sight as well as finding a position for the next waypoint of a vehicle), this can be very helpful in conjunction with BIS_fnc_findSafePos that you tried in your modified script. The nice thing about BIS_fnc_findSafePos is that it already allows us to find a completely random position between some min and max range, so all the code checking for this can be removed. Then we just have to try and find the closest road to that random position and are good to go, without exhaustively collecting all roads in a huge area.

If I'm not mistaken, there are still some issues in your modification, but the general idea is on point. I'm going to try and work on it some more to finally see if it helps performance as much as I hope.

[invrecon]

I probably won't be able to finish the evaluation today. I just noticed that there are a couple more usages of nearRoads <largeValue> in the codebase. Some of them don't really have an impact because they're only executed once at the start, but others might also be worth improving. Examples are fn_FindSpawnRoad.sqf or fn_MoveVehicle.sqf. My current plan is to define a reusable function that uses the combination of BIS_fnc_findSafePos and BIS_fnc_nearestRoad to search for suitable road segments (both for waypoints and spawning vehicles) and replace most of the frequently used nearRoads occurrences with it.

It's definitely worth a try, because finding a random road segment with

_seg = player nearRoads 2000;
_result = selectRandom _seg;

takes 5.27368 ms while doing the pretty much equivalent

_ranPos = [player, 0, 2000, 0, 0] call BIS_fnc_findSafePos; 
if (count _ranPos == 2) then { 
    _roadSegment = [_ranPos, 200] call BIS_fnc_nearestRoad; 
    if (!isNull _roadSegment) then { 
        _result = _roadSegment; 
    }; 
};

is only 0.193573 ms

[dookiejones]

But doesn't it require server, clients and HCs to run the addon?

Werthles can be used as a drop in script as well, modules not required. IHMO the script version is much easier to use, it is pretty much set it and forget it. In init.sqf [true,30,false,true,30,3,true,[]] execVM "WerthlesHeadless.sqf"; WerthlesHeadless.zip

The nice thing about BIS_fnc_findSafePos is that it already allows us to find a completely random position between some min and max range, so all the code checking for this can be removed.

It also allows blacklisting positions in the format [center, radius] The checks for too near another vehicle can also be moved into this function.

[CaptainPStar]

My current plan is to define a reusable function that uses the combination of BIS_fnc_findSafePos and BIS_fnc_nearestRoad to search for suitable road segments (both for waypoints and spawning vehicles) and replace most of the frequently used nearRoads occurrences with it.

Keep in mind that those functions are not inengine function and do basically the same logic as the script does right now (checking if something is between min/max etc). I doubt they are faster than the script right now, when they are used to replicate the current state.

At this point I will rather rewrite the whole traffic stuff completly. I am not very satisfied with the garbage collection and stuff and I want to outsource the AI logic to their own functions.

[invrecon]

Werthles can be used as a drop in script as well, modules not required.

Excellent, sounds like it's a really good option. I was surprised in general how easy it is to set up HC, even with my crappy attempt. It's almost as simple as moving ownership of a unit after spawning and everything else still works (as far as I can tell) thanks to the engine's synchronization magic.

Keep in mind that those functions are not inengine function and do basically the same logic as the script does right now

Good point. What makes a difference is not the functions we use, but the way we use them. Selecting a random position in a 2 km radius and then taking the closest road segment to that within a 200 m radius is simply faster because it only results in a nearRoads 200 instead of nearRoads 2000. A full rewrite would of course be super cool to see (especially for the potential of outsourcing AI logic), but I think I'm still going to try this new method just to see how it works.

[dookiejones]

I am going to investigate this on a server without HC but I am finding village patrols are broken after enemy contact. It appears new waypoints are given but the AI does not respond and it looks like the waypoint logic is starting to spam it's self.

[dookiejones]

Patrols are successfully resuming after enemy contact without HC.

[invrecon]

We should probably split the whole military traffic discussion into a separate issue at this point 😄

So I did what I said I wanted to try, which is basically

1. Build a function that selects a random position and returns the closest road segment within 200 m
2. Use this in the military traffic script to create waypoints and find spawn segments

   
   diff --git a/Code/functions/DRN/fn_FindRoadSegment.sqf b/Code/functions/DRN/fn_FindRoadSegment.sqf
   new file mode 100644
   index 0000000..99c4cd6
   --- /dev/null
   +++ b/Code/functions/DRN/fn_FindRoadSegment.sqf
   @@ -0,0 +1,24 @@
   +private ["_isOk", "_tries", "_result"];
   +params["_center", "_minDist", "_maxDist"];
   +
   +_isOk = false;
   +_tries = 0;
   +_result = objNull;
   +
   +while {!_isOk && _tries < 5} do {
   +    // Find a position with no minimum clearance (we don't care) on land
   +    // On success this returns an [x, y], otherwise an [x, y, 0]
   +    _ranPos = [_center, _minDist, _maxDist, 0, 0] call BIS_fnc_findSafePos;
   +    // Check if we succeeded
   +    if (count _ranPos == 2) then {
   +        // Now find the nearest road segment within 200 m (fine for most maps)
   +        _roadSegment = [_ranPos, 200] call BIS_fnc_nearestRoad;
   +        if (!isNull _roadSegment) then {
   +            _result = _roadSegment;
   +            _isOk = true;
   +        };
   +    };
   +    _tries = _tries + 1;
   +};
   +
   +_result
   diff --git a/Code/functions/DRN/fn_MilitaryTraffic.sqf b/Code/functions/DRN/fn_MilitaryTraffic.sqf
   index 5f9b1bd..342a631 100644
   --- a/Code/functions/DRN/fn_MilitaryTraffic.sqf
   +++ b/Code/functions/DRN/fn_MilitaryTraffic.sqf
   @@ -86,8 +86,7 @@ if (isNil "drn_fnc_MilitaryTraffic_MoveVehicle") then {
            _destinationPos = + _firstDestinationPos;
        }
        else {
   -            _roadSegments = _vehicle nearroads 2000;
   -            _destinationSegment = selectRandom _roadSegments;
   +            _destinationSegment = [_vehicle, 0, 2000] call drn_fnc_findRoadSegment;
            _destinationPos = getPos _destinationSegment;
        };

@@ -115,81 +114,19 @@ _possibleVehicles = [];

_fnc_FindSpawnSegment = { private ["_referenceUnits", "_minSpawnDistance", "_maxSpawnDistance", "_activeVehiclesAndGroup"];

• private ["_roadSegments", "_roadSegment", "_isOk", "_tries", "_result", "_spawnDistanceDiff", "_refPosX", "_refPosY", "_dir", "_tooFarAwayFromAll", "_tooClose", "_tooCloseToAnotherVehicle"];

• private ["_result"];

  _referenceUnits = _this select 0; _minSpawnDistance = _this select 1; _maxSpawnDistance = _this select 2; _activeVehiclesAndGroup = _this select 3;

• _spawnDistanceDiff = _maxSpawnDistance - _minSpawnDistance;

• _roadSegment = "NULL";

• if(count _referenceUnits == 0) exitwith {"NULL"};

• if(count _referenceUnits == 0) exitwith {objNull}; private _refUnit = objNull; _refUnit = vehicle (selectRandom _referenceUnits);

• if(isNull _refUnit) exitwith {"NULL"};

• _isOk = false;

• _tries = 0;

• while {!_isOk && _tries < 5} do {

• _isOk = true;

• _dir = random 360;

• _refPosX = ((getPos _refUnit) select 0) + (_minSpawnDistance + _spawnDistanceDiff) * sin _dir;

• _refPosY = ((getPos _refUnit) select 1) + (_minSpawnDistance + _spawnDistanceDiff) * cos _dir;

• _roadSegments = [_refPosX, _refPosY] nearRoads (_spawnDistanceDiff);

• if (count _roadSegments > 0) then {

• _roadSegment = selectRandom _roadSegments;

• // Check if road segment is at spawn distance

• _tooFarAwayFromAll = true;

• _tooClose = false;

• {

• private ["_tooFarAway"];

• _tooFarAway = false;

• if ((vehicle _x) distance (getPos _roadSegment) < _minSpawnDistance) then {

• _tooClose = true;

• };

• if ((vehicle _x) distance (getPos _roadSegment) > _maxSpawnDistance) then {

• _tooFarAway = true;

• };

• if (!_tooFarAway) then {

• _tooFarAwayFromAll = false;

• };

• _tooCloseToAnotherVehicle = false;

• {

• private ["_vehicle"];

• _vehicle = _x select 0;

• if ((getPos _roadSegment) distance _vehicle < 100) then {

• _tooCloseToAnotherVehicle = true;

• };

• } foreach _activeVehiclesAndGroup;

• } foreach _referenceUnits;

• _isOk = true;

• if (_tooClose || _tooFarAwayFromAll || _tooCloseToAnotherVehicle) then {

• _isOk = false;

• _tries = _tries + 1;

• };

• }

• else {

• _isOk = false;

• _tries = _tries + 1;

• };

• };

• if (!_isOk) then {

• _result = "NULL";

• }

• else {

• _result = _roadSegment;

• };

• if(isNull _refUnit) exitwith {objNull};

• _result = [_refUnit, _minSpawnDistance, _maxSpawnDistance] call drn_fnc_findRoadSegment;

  _result }; @@ -221,27 +158,26 @@ while {true} do { _spawnSegment = [_referenceUnits, _minDistance, _maxSpawnDistance, _activeVehiclesAndGroup] call _fnc_FindSpawnSegment;

   // If there were spawn positions

• if (str _spawnSegment != """NULL""") then {

• if (!isNull _spawnSegment) then { _tries = 0; _refPos = position _spawnSegment; // Get first destination

• _roadSegments = [];

• while {count _roadSegments == 0} do {

• _destinationSegment = objNull;

• while {isNull _destinationSegment} do { _trafficLocation = floor random 8; switch (_trafficLocation) do {

• case 0: { _roadSegments = ([(_refPos select 0) + _goToDistance, (_refPos select 1) + _goToDistance] ) nearRoads 1500; };

• case 1: { _roadSegments = ([(_refPos select 0) - _goToDistance, (_refPos select 1) + _goToDistance] ) nearRoads 1500; };

• case 2: { _roadSegments = ([(_refPos select 0) + _goToDistance, (_refPos select 1) - _goToDistance] ) nearRoads 1500; };

• case 3: { _roadSegments = ([(_refPos select 0) - _goToDistance, (_refPos select 1) - _goToDistance] ) nearRoads 1500; };

• case 4: { _roadSegments = ([(_refPos select 0), (_refPos select 1) + _goToDistance*1.414] ) nearRoads 1500; };

• case 5: { _roadSegments = ([(_refPos select 0), (_refPos select 1) - _goToDistance*1.414] ) nearRoads 1500; };

• case 6: { _roadSegments = ([(_refPos select 0) + _goToDistance*1.414, (_refPos select 1)] ) nearRoads 1500; };

• case 7: { _roadSegments = ([(_refPos select 0) - _goToDistance*1.414, (_refPos select 1)] ) nearRoads 1500; };

• case 0: { _destinationSegment = [[(_refPos select 0) + _goToDistance, (_refPos select 1) + _goToDistance], 0, 1500] call drn_fnc_findRoadSegment };

• case 1: { _destinationSegment = [[(_refPos select 0) - _goToDistance, (_refPos select 1) + _goToDistance], 0, 1500] call drn_fnc_findRoadSegment };

• case 2: { _destinationSegment = [[(_refPos select 0) + _goToDistance, (_refPos select 1) - _goToDistance], 0, 1500] call drn_fnc_findRoadSegment };

• case 3: { _destinationSegment = [[(_refPos select 0) - _goToDistance, (_refPos select 1) - _goToDistance], 0, 1500] call drn_fnc_findRoadSegment };

• case 4: { _destinationSegment = [[(_refPos select 0), (_refPos select 1) + _goToDistance*1.414], 0, 1500] call drn_fnc_findRoadSegment };

• case 5: { _destinationSegment = [[(_refPos select 0), (_refPos select 1) - _goToDistance*1.414], 0, 1500] call drn_fnc_findRoadSegment };

• case 6: { _destinationSegment = [[(_refPos select 0) + _goToDistance*1.414, (_refPos select 1)], 0, 1500] call drn_fnc_findRoadSegment };

• case 7: { _destinationSegment = [[(_refPos select 0) - _goToDistance*1.414, (_refPos select 1)], 0, 1500] call drn_fnc_findRoadSegment }; }; };

• _destinationSegment = selectRandom _roadSegments; _destinationPos = getPos _destinationSegment;

       _direction = ((_destinationPos select 0) - (getPos _spawnSegment select 0)) atan2 ((_destinationPos select 1) - (getpos _spawnSegment select 1));

  diff --git a/Code/include/functions.hpp b/Code/include/functions.hpp index 8995a2c..5b50ae9 100644 --- a/Code/include/functions.hpp +++ b/Code/include/functions.hpp @@ -173,6 +173,7 @@ class CfgFunctions class DepopulateAquaticPatrol {}; class InitAquaticPatrolMarkers {}; class GarrisonUnits {};

• class FindRoadSegment {}; }; }; class ATR

  
  As you can see it massively simplifies finding a spawn segment, but at the cost of not doing the exact same thing. For example, it doesn't check whether it's too close to other vehicles or units (not using the blacklist parameter that `BIS_fnc_findSafePos` offers). But it's a good enough approximation for my testing.

While trying it I haven't seen it introduce any issues. Vehicles are still being spawned at the same rate as before and I tend to encounter about the same number of them while driving around. And I'm happy to report that at least on the old and inadequate computer I use as a dedicated server, performance is improved quite a bit. While spawning a new vehicle would bring down the server FPS to below 10 for several seconds before, they're now solidly around 30 while spawning and drop for a much shorter time before climbing back up to 40+.

[invrecon]

And combining it with the simple 4-line HC offloading I mentioned before, I get an average of 47 server FPS while driving around the map, it's amazing.

[dookiejones]

From what I have been able to find fn_Patrol.sqf is breaking after locality change. Changing spawn on line 35 to remoteExec continued to work but still failed after locality change. I was unable to get the markers on line 16 working with remoteExec

[dookiejones]

if(!isserver && hasInterface) exitwith {}; Got patrols working as intended, though debug markers for patrol path are broken.

[dookiejones]

Players have double addAction entries for revive.

[dookiejones]

I did some stress test runs with 6 players,CUP, Altis, Chernarus summer and winter, Nogova, United Sahrani, Takistan, Livonia, AI skill normal, Amount "A lot", War torn on, Village patrols high, Latest DEV build of Escape, plus my own tweaks.

Village patrols on high absolutely TANK, single digits, server FPS even with 3 HCs connected. This ended up at roughly 160 AI per HC in larger areas or near multiple small towns. I am assuming this has to do with patrol markers and way points as server FPS did not increase with AI offload to HC in the highest pop areas.

A check should be added to prison location to keep it away from village markers, having a whole villages AI run up on you because you opened the prison doors is fun. Perhaps just add more mags to the backpack!

I did not have time to dig deep but the double action menu items for revive are a bit odd, I would expect 4 entries if it were the HCs, possibly correctable with if(!isserver && hasInterface) exitwith {};

I never noticed significant mortar fire but there were a few explosions that could have been, we were actively trying to avoid getting pinned and shelled.

Other than mild server FPS issues that were most likely caused by the settings, I normally run "a lot" and patrols medium with war torn on, the mission seemed to go off without a hitch. Sightings were acted upon, reinforcements were called, Everything seemed to spawn and act as it should. The choppers even landed smooth as butter.

[invrecon]

I did some stress test runs with 6 players

You can test the mission with 6 players? That's fantastic, I wish more of my friends were that enthusiastic about ArmA! Would you be willing to try my version too? It's based on your work for the military traffic and it would be super valuable to hear whether you think the spawning is still close enough to the original and if you can also observe better performance. I'd love to join you for testing, but from your posting activity I'm guessing you're in a US timezone while im in Europe. So scheduling would be difficult and the ping would also be an issue.

co10_Escape_Contact_CSAT_vs_NATO_HC.Enoch.pbo.zip has the military traffic changes and also uses 1 HC for ambient infantry and units involved in military traffic. You probably shouldn't stress this one as much as you did yours, because it's not as much about HC, but about the vehicle spawning. Although it would allow you to check if my way of using HC causes any of the issues you've been seeing with Werthles. It's on Livonia, but if you'd prefer to build it yourself (for other terrains too) I could make a fork and apply my changes.

It would be particularly interesting to compare it to the original version (built from master branch) co10_Escape_Contact_CSAT_vs_NATO.Enoch.pbo.zip

It's not necessary to test it with a significant number of people, it's enough to go in alone, spawn a vehicle and start driving around to see the difference in server FPS (at least on my system).

[dookiejones]

@invrecon Sometimes I can get the mission full enough that I want to add more player slots. I have 2 Escape servers up, one is vanilla minus Tanoa since I lose most of the server pop when Tanoa come up in rotation, The other is CUP with my own tweaks. Search for Waffle House, discord is in server title, and see what your ping is, the server is US eastern time zone as well as most of the players that join discord. Ping may not be an issue for you. I am more than willing to have you join us for testing or play, the servers are open to the public. I have plenty of hardware sitting around so I can spin up whatever you want Arma wise, 2 servers and 6 HCs currently on dual Xeon box so I certainly have space for more. My posting times these last couple days have been odd for me, I work for myself and took a few days off. I am normally on during the evening EST. PM me on discord, same name on the Map builder and escape discord, and we can try to setup some time for testing. I am sure most of the guys would join us for testing if we set it up for a weekend. Once I get home this evening I will run through the Livnoia missions you posted and get back to you here.

Do you happen to know what file contains the init for the revive system? I cannot seem to find it and I would love to fix the double action menu entries.

[invrecon]

Cool, looking forward to it. I just tried joining the server and ping really does seem fine.

Revive should be in Revive\functions\Revive\fn_ReviveInit.sqf.

[dookiejones]

@invrecon I did some testing with the missions provided. Everything at stock settings. No HC mission ran just fine, lowest server FPS I saw was 33. The HC mission the lowest server FPS I saw was 43 but it mostly stayed in the high 40s.

I changed the AI group size to large, spawn distance to far, village patrols to high, war torn on and as expected server FPS dropped as low as 6.

All the above settings with village patrols on medium lowest server FPS was 10.

[invrecon]

Excellent, thanks a lot! It was a bit dumb of me to combine HC and the military traffic tweaks in one mission, so we can't really tell which gives the greater benefit. But if you didn't notice anything off about the vehicle spawning in the HC mission, that means at least it doesn't break anything, which is also great to know. And of course that the HC doesn't cause any weirdness like the duplicated revive options you've been seeing with Werthles.

[dookiejones]

@invrecon I was not able to test with another player to see if there were duplicated revive options or not. I did forget to mention that the debug markers for patrols are still broken with your version.

EDIT The debug markers are very unreliable for patrol paths, sometimes they do not update, sometimes they do seemingly at random.

[invrecon]

The revive should be fine, I tried that with a friend earlier today. Too bad about the debug markers though.