Improve AI

[dnk777]

I have a plan to significantly improve AI. About 20% or planned changes are made, and progress will depend heavily of my load at work, but current changes are already worth to see. My code is in very early stage, and contains well-known bugs (a memory corruption source of what I did not spot yet), but I cant wait to share progress. The code is in development in my fork.

Done:

• Bots never did bunnyhopping at least in 1.5+, fixed.
• Old bot area checks were stupid and caused bots getting blocked in walls (AI_CheckMove just checked a trace from eyes to forward ground, and detected walls as a ground).
• Old bots have dumb target and weapon selection, they used stick to a last attacker. My code uses quite sophisiticated EnemyPool, that tracks seen enemies, last targets and attackers, suggests targets and weapons using complex heuristics based on distance, health status, powerups, direction, gravity and environment, and suggests advance/retreat/inhibit fight tactics. Bots now do weapon combos to finish enemies. Bots now try to spam last seen enemy positions if there is no enemy in view.
• Old bots never evade projectiles due to a bug. Bots now may evade rockets, gunblade blasts, grenades, plasma streams and laser beams. Plasma projectiles are clusterized in rays based on direction and proximity. Bots evade and dash heavily in fight.

Short-term goals:

• Fix my bugs :)
• Introduce more human-like hitscan aiming by calculating speed of change of dot products between look dir and dir to enemy.
• Indroduce "true" strafe-jumping as described in "Strafing theory" by injx, its a 5-minute task itself but depends of fixing bot mouse movement (bots roll mouse like a drunk each second atm).

[dnk777]

I will also try the master branch again later today to see if it's any better than the aas-port branch.

This means that you played working (that does not crash) but unplayable branch. Master is two weeks of bugfixes ahead of it.

[viciious]

Just tried the master branch and failed to notice any difference: bots still jump around the corners, aimlessly turning their heads into random directions. At first I thought it was my buggy bspc producing bad AAS files for warsow maps but stock q3a maps exhibit the same bot behavior.

[dnk777]

The updated branch is there: https://github.com/dnk777/qfusion/tree/dnk777_rebased

I do not know what you mean complaining about bot behavior. Its of course weird but is a good point to start improve it for me. http://webmshare.com/jjD1g

Most of AAS files produced by BSPC can't be loaded. I tried generating AAS for wdm1, wdm7, and it fails with something like this:

Gametype 'Free for all initialized'
--- Botlib initialization ---
----- Map loading -----
trying to load maps/wdm1.aas
**********
ERROR: Game Error:  G_Z_Malloc: failed on allocation of 230193 bytes
**********
==== G_Shutdown ====

wdm9 though works fine. I did not try loading any Q3 map yet.

[viciious]

I'm not complaining, just pointing out. Try speccing a lone bot to see him suffer through the map.

I tried generating AAS for wdm1, wdm7, and it fails with something like this:

In g_local.h, change G_LEVELPOOL_BASE_SIZE to 15MB

[dnk777]

To have not-looking-weird bot behavior I have to implement seamless transition between combat and roaming movement. Roaming movement is planned via AAS subsystem. Combat movement is random and only affected by "attack"/"retreat" flags in CombatTask, it is not aware neither of map topology, not of goal entity origin. Bot should not lose goal in combat movement, but to implement it I have to make much more sophisticated goal planning with cancellation of goals (when bot is weak) and the enemy blocks it.

That is a recent video of "camping a spot" movement which is not used atm (I forced it for debugging purposes in private build), but is very useful for waiting for goal pickup (item timing is a part of goal subsystem improvement and will be added soon). http://webmshare.com/jjbLO The code that produces this kind of movement is here: https://github.com/dnk777/qfusion/commit/b28f8dc14d68e679ed25f881c0296ebee0712760 The video shows "camping a spot" movement with different parameters (alertness and intended look-at-point, specified or not). Imagine a bot doing this waiting for armor pickup/bomb plant.

[viciious]

Imo improving combat or camping AI should be of lesser concern at this point... http://webmshare.com/DA5XG

[dnk777]

Half of these bugs I guess were fixed last day, and roaming movement will be improved anyway for the purpose of combat and roaming movement fusion. And that "camping a spot" movement is very useful not only for combat/goal actions, but for roaming movement too (a bot will move this way on an elevator platform).

[dnk777]

How do you look at a bot cheating with movement, if it is done carefully and not very-robotic looking? I mean a bot that has an ability to set both magnitude and direction of self->s.velocity while in air (self->groundentity == NULL, ground/combat movement is to be kept as is)? The problem is with trying to control a bot that tries to follow usually curved AAS reachability chains with an ordinary movement keys and thus fully affected by the player physics with inertia. You saw that even making a bot to pick an item is quite complicated due to control inertia. I know that the praised CPMA/Spiterbots that are able to imitate a high-skilled in movement player cheat with movement. Breaking client-side player prediction is the downside of this approach.

[viciious]

That should be the last resort.. As of now, I can't see how the problem lies with the aircontrol inertia. The problem as I see it, is getting bots to travel from point A to point B. Enabling bunnyhopping movement prior to getting the very basic things to work was a mistake. Bots have a hard time picking up items even if they are in the same AAS area, even if it would require walking in a straight line, which they attempt and just walk past the item very often.

[dnk777]

Bots have a hard time picking up items even if they are in the same AAS area,

Was fixed in https://github.com/dnk777/qfusion/commit/c4ba52439983ee9f64dd05e7ca8d8054705fb197.

The problem is in areas size, they are usually too small and are changed too often during fast movement. An area is a simple convex volume determined automatically. Where a human sees a large room, e.g in a quad-to-tele area on wdm9, bot has lots of small convex areas connected by reachabilities that are determined by bspc too. I'd rather prefer navigation volumes (not nodes, volumes are much, much superior due to simplicity of position/reachability checks!) that are placed manually by a mapper but writing an editor for it is not a rewarding thing. The situation is worse on the maps with more complex collision geometry, they behave not so bad on wdm9. I either try to follow AAS reachabilities chain, which is rarely straight and for small areas require lots of control in movement or try to straighten the path through areas as long as a straightened path follows these areas (this code is now turned off due to amount of movement mistakes it introduces; there is no a cheap way to check whether straightening a path/shortcutting is legal).

I'll try it in there weeks... good players are able to do some «supernatural» moves anyway. I believe a bot that moves fast and precise but, maybe, a bit unnatural is better that a bot that cannot bunny. Is misleading the netcode prediction the only thing that may have a negative impact? And netcode prediction impact is to be proven yet, considering relatively high server rates the game runs on.

[viciious]

Was fixed in dnk777@c4ba524.

No it wasn't. On, say, wdm1, bots miss the quad and ra in like 70% of cases, they just walk or jump past by. Overall, their navigation around wdm1 is so bad it actually hurts.

The way they manage to miss the damn quad or the ultra on wdm9 (sometimes two or three times in succession) is equally amusing.

The way they manage to dodge the damn quad on wdm9 multiple times in a row is also pretty amusing.

And netcode prediction impact is to be proven yet, considering relatively high server rates the game runs on.

You could enable the pmove flag that disables prediction for bots (grep for PMF_NO_PREDICTION). That would fix it for specs, at least.

But trust me, you would not want to play with movement prediction turned off. Try cg_predict 0 and connect to any server.

[dnk777]

Does the game use enemy ucmd in prediction of enemy movement for a client? If it just (linearly) extrapolates its entity params prediction errors should be negligible considering high server rate. I do not mean turning off movement prediction for an active POV itself!

[viciious]

Ah, ok, it should be look/work fine then. You still going to need PMF_NO_PREDICTION for bot clients though :)

[dnk777]

On the video http://webmshare.com/K7oBV these cyan lines correspond to AAS reachabilities (aas_reachability_t::start, aas_reachability_t::end) from some area to another one.

for (int i = 0; i < aasworld.reachabilitysize; ++i) {
   auto &reach = aasworld.reachabilities[i];
   if (random() < 0.01) // events flood protection
      // spawn a laser event from reach.start to reach.end
}

They are shown as server-side events and due to limited events count they blink. You can see that each stair step (and even a part of a step) is treated as a separated area. Writing a code that tries to change intendedLookVec by either following these reachabilities or area centers, or short-cutting and straigtening the path is complicated. The March version that looked satisfactory was developed on low-poly Q3 maps. I should develop a code that tries to short-cut this mess instead of following each reachability.

[viciious]

Try different bspc settings maybe?

[dnk777]

I should redo intendedLookVec calculation anyway. Only area centers/faces (the problem with faces is their absence in optimized/stripped AAS files, loading these Q3 maps should not be supported) that are volumetric should be used instead of reachabilities that are a single line segment that consist of two points mentioned above and thus lead to curved movement path and return us to a node/point reachability hell.

[dnk777]

I have fixed most annoying movement bugs, but bot movement still is not more than mediocre. Bots should perform better on simplistic, low-collision-poly maps.

Look at the video I have posted days ago, maps are full of junk areas. AAS compiler treats each horizontal surface as a navigable area, you may notice on the video that even top of the railings may be involved in pathfinding. These small areas can't be easily removed, because it may break map topological connectivity. Bot navigation is a mapper concern too, from what I read in bspc readme a mapper may prevent navigation in some areas by marking parts of a map as «do not enter».

I will try to tweak preferred and allowed travel flags used in pathfinding to cut off these kinds of reachabilities that often go along with junk areas.

[dnk777]

I have implemented seamless combat/roaming movement and goal rejection/cancelation when a bot should retreat and path to a goal is blocked by enemy. These improvements are shown on the video.

http://webmshare.com/bLOwb

Those pink lines on video are drawn from bots to their blocked goals (not rejected by cheaper tests yet). Notice that the just respawned bot on video had enough health to attack first and not retreat, so these lines were not drawn.

[viciious]

Failed to see any pink lines in the vid above even after 3 watch-throughs.

Please also post videos showcasing the actual AI behavior from bots' POV. Vid's recorded from human's POV don't provide enough information on the quality of AI's movement or intellectual skills.

Also noticed while testing that the way bots use pg and lg is completely unrealistic (aimbot precision individual shots) and they switch between weapons waay too often, even in close combat. And ye, the way they move is still completely retarded.

[dnk777]

Failed to see any pink lines in the vid above even after 3 watch-throughs.

These lines may be seen in the second part of the video, they are drawn from bot origin to 50hp each time when bot evaluates goals and finds that he has to retreat to some goal and path to the 50hp goal is blocked by enemy

the way bots use pg and lg is completely unrealistic (aimbot precision individual shots)

Its tweakable, its very minor issue

they switch between weapons waay too often

It is indended, when weapon selection code is crafted well enough they should select appropriate weapon according to constantly changing combat disposition.

And ye, the way they move is still completely retarded.

Did you check the last commit?

[viciious]

Now I have. Indeed things have gotten somewhat better now, although bots's aim still wobbles too much and they still love spinning and circling around, as well as aiming straight up. wdm1 is still a very problematic map for them.

Also noticed that due to bot movement circumventing regular player movement, they make no sounds whatsoever.

[dnk777]

Also noticed that due to bot movement circumventing regular player movement, they make no sounds whatsoever.

I change self->velocity only in air, and the bot as any player moves silently in air anyway, all dash/jump sounds should not be skipped. All jumps/dashes/ground movements are made as any other player do it.

[viciious]

Could have been a glitch but I can swear I couldn't hear any bot sounds at all.

On the movement side, the few remaning major problems are:

• bots bumping into walls a lot
• bots walking past goal items (wdm1 ra is a good example)
• occasional poor navigation even in good conditions: for example, once bot got stuck in ga room on wdm1
• support for teleporters, doors, jumppads and platforms

[dnk777]

bots bumping into walls a lot

They do not know how to walljump yet and just try to avoid walls testing by tracing in runtime, and aircontrol that is not obviously cheating is too weak in this case :) Should be mitigated when walljumping is implemented.

bots walking past goal items (wdm1 ra is a good example)

Frankly THIS item may be problematic due to weird AAS reachabilities. It may sound discouraging but mapper support is still needed, some junk areas that possibly break path should be marked as «do not enter». Lots of junk areas behind wdm9 pipes near tele-UH are an example. Can't do anything with it.

occasional poor navigation even in good conditions: for example, once bot got stuck in ga room on wdm1.

Please describe where exactly it has occurred, I have to know is it possible too sensitive runtime obstacles check or just reachabilities glitch.

support for doors, jumppad and platforms

Jumppad movement should not have very much issues. I have visualized this version of jumppad movement in https://github.com/dnk777/qfusion/blob/dnk777_rebased/source/game/ai/bot_movement.cpp#L312 by drawing a line from a bot to a chosen area center and did not found any glitches while testing.

Platforms are handled there https://github.com/dnk777/qfusion/blob/dnk777_rebased/source/game/ai/bot_movement.cpp#L449, but it may not work as expected. Look at the video above I have posted showing wdm1 reachability mess. There are lots of junk areas on the YA platform that are involved in pathfinding, and when the platform enters TOP state, he possibly tries to follow there areas.

Door support usually depends of AAS pathfinder. On wdm7 bots never chose an obvious path to quad via door, and run around the lower level to reach it. On some other maps, e.g. wbomb2 doors are handled as non-solid things. Door buttons handling is not implemented (and was not in old code).

I agree that movement on wdm1 is problematic, they behave better on 7/9.

[dnk777]

On frequent weapon changing, I took care about preventing weapon choice «jitter» those days (bots were changing weapons all the time and not shooting due to weights that are close and changing on each weapon choice reevaluation). https://github.com/dnk777/qfusion/blob/dnk777_rebased/source/game/ai/bot_brain.cpp#L1135

[viciious]

Please describe where exactly it has occurred, I have to know is it possible too sensitive runtime obstacles check or just reachabilities glitch.

A bot spawned in the ga room on wdm1, decided it'd go for the mega and then started circling around the room: riding on the platform and then dropping back.

Jumppad movement should not have very much issues.

wdm1 jumppads do cause troubles for bots, especially the one that leads to the ra and eb ammo.

frankly THIS item may be problematic due to weird AAS reachabilities.

Dunno.. Bots frequently drop off the ra platform by turning 90 degrees away from the goal and walking or jumping off the ledge. I very much doubt AAS reachibilities promote such paths so close to goal items.

[viciious]

On the subject of jumppads, try wda5: bots fail to climb the longer jumppad 9/10 times and also tend to look straight up while in the air.

[viciious]

It is indended, when weapon selection code is crafted well enough they should select appropriate weapon according to constantly changing combat disposition.

Well, as a matter of fact, they should not. In real combat there's simply no time to switch between your weapons, deferring the shot. In real duel combat there's only like two weapons: the primary weapon and the secondary finishing weapon. None is based on the distance, rather more the amount of ammo you have and your estimate of opponents health and armor.

At the moment, bots consider constantly switching their weapons more important than actually shooting them.

[dnk777]

I discovered quickly those days why bots fall in places like wdm1 RA (RA item «magnet» starts working prematurely when an item coarse radius (~60-90 units) is reached without checking a ground below a bot, it should start only when a bot is in item AAS area or AAS reachability chain contains only ones of movement type TRAVEL_WALK), but had not enough time to implement and check it to avoid claiming the thing as fixed. The development will be continued soon in 1-2 weeks.

[dnk777]

None is based on the distance, rather more the amount of ammo you have and your estimate of opponents health and armor.

Weapon selection is not based on (only) distance, but on many things. A sophisticated weapon selection code is one of the first things I crafted started working on the project.

At the moment, bots consider constantly switching their weapons more important than actually shooting them.

Its tweakable by modifying a threshold of difference between suggested and current weapon weights. Also, I have added a hack (increased threshold) for continuous fire weapons. https://github.com/dnk777/qfusion/commit/9504d786512c5f20054523166684844aed318769#diff-c0beaa16909d7ce9bec5826781605a25R1362

it should start only when a bot is in item AAS area or AAS reachability chain contains only ones of movement type TRAVEL_WALK

Fixed in https://github.com/dnk777/qfusion/commit/9b3c42ed45c11b605c8678777b3169e66121d3ac, its unlikely a bot will fail picking up an item like wdm1-RA.

On other movement glitches… I was in hope that AAS will provide a reliable way to handle bot movement, but the code entirely consists of dirty hacks that sometimes fail. I think if bot has troubles in some areas, a mapper should forbid these areas by marking parts of a map as «do not enter» as said in bspc readme.

What the map compiler produces is a junk, nowhere close to a sane map topology (rooms and hallways) what a human can see and I expected to introduce… It is partially caused by a complexity of the game collision surfaces compared to Q3 ones. These tiny areas that a bot has to navigate through and that may be passed in a single frame only confuse a bot. To mitigate this some kind of weighted interpolation is used by me, and it sometimes may fail. On these videos areas in radius of 48 units around a bot are shown randomly sometimes (to prevent beam events flooding), often it is a complete mess. Note that areas are not boxes but convex objects, their bounding boxes are shown.

http://webmshare.com/YxnA5 http://webmshare.com/xgnZg

[viciious]

Have you tried compiling .aas files with the -optimize setting?

[dnk777]

An -optimize option does nothing but strips AAS file leaving only area AABBs and reachabilities, and that should not be done (AAS geometry such as faces and planes are used, e.g., for target environment sampling for RL aiming providing better quality and/or performance than tracing). AAS area merging is always performed (as it may be seen in logs), but it does not help much as we can see.

https://github.com/TTimo/bspc/blob/master/bspc.c#L819 https://github.com/TTimo/bspc/blob/master/deps/botlib/be_aas_optimize.c#L26

[dnk777]

I have finally started a work on the teamplay logic. This video shows squads clustering and reevaluation.

http://webmshare.com/VzV8Z

Lines are drawn from time to time for each pair of squad members (a squad consists of 2-3 bots that should roam and fight together). On 0:15 links reevaluation is clearly visible. They don't try to stick and do actions together, but already share enemies and notify each other. Common squad behaviors such as «assist item pickup», «assist a carrier», «prevent item pickup», «plant an item» will be implemented and made available for GT scripting. Weapon dropping will be added too.

Also, that's a funny video that shows forced backward movement for a bot.

http://webmshare.com/D3eKQ

Obviously a last bot in a squad should move this way to cover back sector when a squad moves together.

[dnk777]

A basic weapon dropping for squad members has been implemented. It of course needs more tweaking. On the first video PE#1 grabbed items dropped by itself (because a suppliant YZ2112 preferred to fight with someone). Weapon drops may be noticed by a blue line drawn between a supplier and a suppliant and a team chat message.

http://webmshare.com/LX77L http://webmshare.com/vEMMD

Also, basic walljumping has been introduced first time even in my branch.

[viciious]

Tbh, bots should only be able to play bomb and duel decently and that's it :P

[dnk777]

Thats not a problem to find a duel/bomb game with human players in the current state of the game, and bots are too stupid for duels anyway. Team-based GTs with items are mine main target, considering how unpopular are they and how carefully crafted bots can aid them by filling empty player slots on public servers.

[dnk777]

I have coded a support of objective-based gametypes (that has defence/offence spots and carriers), bots in a team receive defend/attack orders dynamically based on an alert level on a spot (bots report to a team when a defended spot is being attacked, and fair bot vision is used, so it is possible to sneak to a defended spot). This gives a foundation for CTF, CTF:T, Bomb gametypes support. GT scripts modification is required because script interface has been changed, so I have forked scripts too.

For now the CTF script is completed, I even tested that on public with bots and real players.

Also I had to significantly modify AAS routing code to support proper obstacles (areas blocked by enemy) handling. Obstacles were handled aposteriori (a path used to be retrieved and than checked for blocked areas), and that lead to poor bot behaviour because a different non-blocked path might exist. This requires having a separate route cache for each bot, because blocked areas status differs for each bot. Because AAS code had to be modified anyway, I just ripped all useful code from former aas subdirectory, its about 2KLoc and removed other code unrelated to a bot runtime. Introducing a separate route cache is absolutely required for proper obstacles/blocked areas handling, even if it hurts performance a lot by far than everything else (when an area status is changed, a routing cache must be almost entirely dropped). In order to mitigate performance issues I optimized basic AAS routing algorithm too by using a (sorted) heap and not just a linked list of area fringe in breath first search, so it helps to reduce count of iterations 5-6 times.

The thing is almost complete (this means there are no more major native code changes planned), but tweaking, testing and bugfixing will take around 1-1.5 month.

Also some minor things should be introduced, such as dropping health/armors for teammates in CTF:T (it may be based on already existing and working code of weapon dropping), adding botroam entities support for item-less gametypes, etc.

For the movement issues, fixing the single one opens a can of worms, so the solution is to handle this per map level via AS api and a mapscript (set limitations of movement capabilities for some areas). This will be very cheap (checking current area custom flags), so it wont hurt performance.

[viciious]

For the movement issues, fixing the single one opens a can of worms, so the solution is to handle this per map level via AS api and a mapscript (set limitations of movement capabilities for some areas). This will be very cheap (checking current area custom flags), so it wont hurt performance.

I don't think that expecting mappers to write AS code for bots is a feasible idea. In my 15 years of working on mods and games I've seen only a few artists who were able to cope with code. And by saying they "coped" with it, I'm being overly generous as most of it was a blind copy&paste hackjob.

Neither I intend to provide backwards compatibility for third-party map scripts..

[dnk777]

I do not mean coding a logic, just declaring movement limitations for some points on maps. A knowledge of a problematic point origin is all that a mapper need.

Something like this in mapscript (not coded yet):

AI::SetAiMovementFlags(Vec3(-1928.0, -213f, -324f), 48.0f, AI_MOVE_ALL_CAPS & ~AI_MOVE_CAPS_DASH);

On this call all areas in radius of 48 units should be found via AiAasWorld::BBoxAreas() and marked as "no-dash". Since a mapscript execution is done once for a map at its loading it wont hurt performance (movement caps are intended to be retrieved by a single array lookup by an area index).

This may be handled on mapping tools level, but 1) It requires modifying of the bsp compiler and the editor. 2) It requires re-compiling a map.

There are usually not more than 2-3 areas on an ordinary map that lead to bot failure. Bot navigation always have been a mapper concern too, and I already tried to minify amount of a mapper job.

[viciious]

1) It requires modifying of the bsp compiler and the editor.

donotenter and botclip flags are still supported both by the compiler and the engine.

2) It requires re-compiling a map.

The opposite also holds true. Modifying brush geometry may require editing the code.

All in all, I strongly suggest you should drop this idea. I never personally liked mapscripts in general. Trust me, possibly using them for bot navigation is bad, bad idea. Fixing poor bot movement with map-specific AS hacks is the road to hell.

[viciious]

Oh, btw, in case you haven't noticed, Warsow development has been discontinued.

[dnk777]

I has an obvious personal interest to continue it by myself.

[viciious]

Btw, have you run performance profiles on your code or tracked its memory footprint/leaks in valgrind?

[dnk777]

I ran AddressSanitizer sometimes (it did not spot any leaks), and used valgrind(cachegrind) for profiling.

As I said, toggling areas enabled/disabled status hurts performance a lot but there are no other reliable ways to avoid blocked areas. When an area disabled status has been changed, almost entire routing cache must be cleaned up. In order to mitigate it a bit I sped up (around 1.5x from my measurements) the routing algorithm itself (drained caches should be populated faster). It used a deque of linked aasroutingupdate_t, and I use a thing similar to priority queue. On wbomb5 in a CTF game main loop of UpdateAreaRoutingCache function referred below often iterated 17000-18000 times, and after change it has never exceeded a limit of 5000 iterations. Maintaining a heap has some additional cost, thats why resulting performance gain is not as large as it was expected. I also used a optimized for access patterns memory layout for oldAndCurrAreaDisabledStatus, inlined ClusterAreaNum manually in the loop and precached all references to global objects as stack-allocated locals to avoid pointer-chasing. https://github.com/dnk777/qfusion/commit/693232e0204196ad638a6eabe1beff4fdc0aeabc

Also I took care about reducing count of routing calls during a search for a goal by selecting not more than 16 best items and testing routes only then in hope the router won't fail for at least a single item. https://github.com/dnk777/qfusion/commit/26e0ff1c92ad9e378da6c6ca00c2b9c24e668c1e

On a listen server on huge maps performance drops are very noticeable. On a dedicated server it should be satisfiable, I have never seen drops below 60 fps on very huge maps in CTF/CTF:T (these gametypes use most complicated bot logic).

Performance have always been an thing I took care about during the development of this code, I optimized features once they were well established many times. The most common optimization pattern I used is to reduce number of expensive trace/routing calls by trying to avoid these calls by doing all cheap rejection tests first, pre-selecting various items lists by filtering and limiting inputs and calling expensive calls only then for selected items instead of doing the calls mentioned above in a loop on a raw list. There are numerous examples of this approach in the code, the most recent is https://github.com/dnk777/qfusion/commit/2af83f62cd5ca5d81a8a4626c86378acdc429372

[viciious]

below 60 fps

How huge was the drop compared to stock Warsow? The minimum acceptable framerate for Warsow is generally considered to be 125fps.

[dnk777]

The drop is entirely caused by code of my bots, and I have mentioned above the main source of performance problems. The performance impact after introduction of a separate route cache for each bot that has an ability to dynamically enable/disable routing areas was noticeable, but bot behaviour benefits a lot from this change, and that is clearly visible. Without proper exclusion of blocked areas from routing bots used to either rush headlessly in a crowd or be blocked in a corner. The thing is in final stage but still is in active development, so I will try to optimize it more. I do not try to advocate myself but any arena game AI that pretend to be «done right» is known as a CPU burner (I am speaking about UT and especially Q3 CMPA bots). On a dedicated server I had a solid gameplay experience, even tested that in public. For low-end clients like Android a separate build that have some CPU-expensive features cut off may be used. Gameplay on Android is different from the vanilla game anyway, and hosting a public game on a phone is a joke, so barely anyone will notice any difference.

[viciious]

Have you considered limiting routing cache resets to only 1 bot per frame? Pretty sure that will not cause any noticable impact on the gameplay. If that doesn't work out, the only other remaining option is be multi-threading, which is something modern games do for the AI anyway. But that would take significant effort.

[dnk777]

Routing cache may be reset not more frequently than 4 frames for sv_fps="60". I took care about uniform distribution of workload from the very beginning by splitting all updates for all ai-related classes in Update() into Frame() (that contains urgent updates that cannot be skipped) and Think(), where Think() is scheduled by some global manager. https://github.com/dnk777/qfusion/commit/7d7aba8f9c760d61b5cdaa68cac914b4e7ad3e14 For example, in ffa game with 8 bots Think() should be called on each frame for 2 bots. In ctf game workload for 8 bots is distributed this way:

frame N+0: Alpha team brain thinks (it may be as expensive as thinking of four bots) frame N+1: Four Alpha bots think frame N+2: Beta team brain think frame N+3: Four Beta bots thinlk

What hurts performance itself, is not the cache resetting itself but re-populating drained cache in following routing calls, so AAS routing should be sped up more than in my attempt https://github.com/dnk777/qfusion/commit/693232e0204196ad638a6eabe1beff4fdc0aeabc.

For a dedicated server multithreading is not an option (because usually even multiple instances of a server are run on the same machine). But dedicated server fps is limited to 60 anyway. I think the common good VPS hardware (2+ GHz) is capable of running 2 CTF games on the same machine (when I tested a server on an unknown VPS it shown 40-70% CPU load).

For a listen server this may be an option, since client updates frequency is vital. AiAasRouteCache ops are not thread-safe, but bots use separate route caches now (because set of blocked areas is different), so updates of routing cache for more than 1 bot that should occur in a single frame may run in parallel in separate threads. All ops that heavy rely on routing are easy to isolate, so an ordinary fork-join pattern may be used.

[viciious]

What hurts performance itself, is not the cache resetting itself but re-populating drained cache

Obviously, but it's the cache drain that causes the re-populating.

All ops that heavy rely on routing are easy to isolate, so an ordinary fork-join pattern may be used.

No, it may be not. Join is going to cause the joining thread to block until the joined thread exits, which is pretty much the same thing as running the job in a single thread. Instead, the worker threads should be assigned jobs, which they would perform in the background and notify the scheduling thread upon completion.

I'm very reluctant to provide the MT functionality in the game module though. Having the AAS routines alone there sucks badly enough already.

For a dedicated server multithreading is not an option (because usually even multiple instances of a server are run on the same machine).

You're confusing physical CPU's with virtual CPU cores/hyper-threading pipelines. Instancing game servers inside docker-alike containers with a variable number of CPU cores is pretty much the de-facto modern day standard.

[dnk777]

Looks like a false alarm for me. Most drops were caused by my client, looks like an occasionally measured debug build or driver issues. I thought you have tested the build and experienced same drops as me, haven't you? On regular maps client fps usually never drops below 200 fps, and even on outstanding by navigation complexity maps wbomb5, wctf2 where I witnessed a bad performance it never goes below 120.

I forgot to say, a routing cache is never cleaned neither each frame, neither each Think(). Updating disabled areas status is tied to combat task selection or update which should be never called more frequently than 200-300 milliseconds.