Any way to benchmark game?

[mirror176]

Describe the feature you'd like Easy access to measure max FPS, CPU load, etc., Maybe a command line launch with different game settings (already there unless needing to expand on it) and reported output though a GUI button/report can be useful too. Could use a prearranged sequence or run on a replay.

Describe why do you think it is needed I tried to test impact of different settings like graphics backend, shadows, etc. and resorted to loading the tutorial and touching nothing to then read FPS (with vsync off) to have at least some kind of baseline Contrary to reports I read, I found OpenGL appears to perform better than Vulkan; checked on GTX570+FreeBSD and GTX660+Windows 10. Maybe it will vary on a more complicated game scene but the starting #s are drastically better for achievable frame-rates. Shadows do have a noticeable impact in that basic case but I find low FPS games (lots of unit+AI activity leading to regular lagspikes unless game was < .33 game speed) didn't help much with disabling them. Art Revolution mod showed that was more drastic but I haven't used it much for gaming yet. A side effect of command line launch+output of results is the game may end with more exposure from things like Phoronix Test Suite and may lead to getting tested against changes in compilers, graphics drivers, operating systems, etc. when those running it find changes worth reporting/publishing. I know I have seen devs here 'somehow' benchmarking their changes but didn't find any steps to do so. A generic benchmark setup that is controllable could likely be easily tweaked to permit generating results that test specific codepaths for them too.

[past-due]

Are you testing with 4.3.5, or the latest master branch?

(The latest master branch includes numerous rendering changes, including support for instance rendering & shadow maps, which should impact performance.)

In regards to benchmarking, a built-in benchmark would be incredibly useful. (We do indeed sometimes manually use replays (or loading savegames) to do performance comparisons when testing PRs.)

[mirror176]

Sorry for not clarifying. Still on 4.3.5 here and haven't done my basic 'tutorial FPS' test for a number of versions. I recently messed around with things enough to get local skirmish mode to launch the wave defense with just me and AI allies. Game goes to some pretty low FPS when looking at many units/buildings or even active battle though looking away restores that from < 30 to > 1000 and disabling shadows does have an impact but its smaller than looking away. That mode leads to very frequent bigger stutters, some shorter to longer freezes, and eventually the game just locks up completely; none of that lag is impacted by shadows or looking away that I could tell so figure its CPU limited by game engine, one or more AIs (I was playing with all different sorts), and/or the wave code and its AI.

As a workaround, any chance devs might share/document guidelines of how benchmarking/profiling has been taking place?

[vaut]

The wave scenario is very heavy and the loads are very different from the rest of the game.

1. the number of units on the map easily exceeds 3k. In normal combat, 1.5k is a limit that is rarely reached.
2. expensive and frequent operations with a script controlled by a bot.
3. UNREALLY large concentration of units in one place. When attacking a landing zone head-on, there can easily be more army on the screen than in a normal battle.
4. Players use a lot of long-range artillery, which is constantly working.

I think long freezes caused by AI scripts, not rendering. Also check that the game has enough memory.

[mirror176]

I'll try to watch memory closer in the future; it was ran on Win10 x64 with 32GB RAM. Loading AIs as bonecrusher started the game with choppy lag and with nullbot started the game running pretty smooth. Harder to get survivability with bot friends when the bots don't reliably play very well though. Last attempts to launch it with last slot set to wave ai on different team leads to an enemy in our spawn location so trying to remember what I am doing wrong about game setup. Wish lobbies had more layout 'fixing' for game modes but that will be a different issue to open.

As for testing without AIs, I still consider myself a 'bad player' compared to my other RTS standards. Played '1' round on autohost with some other people who seemed friendly and just wanting more players instead of the 'you have to play >=me to be in game with me' attitude I so often see in bigger commercial game player bases. Sadly I'd feel a bit more embarassed 'experimenting' with different defense ideas while not against AI as I may try things that make no sense for experienced players so sometimes would like it in private. Thanks for the game mode.

Edit: Don't recall the lag during the autohost game without AI allies but was distracted just trying to control my player in game while watching+learning.

[vaut]

I believe that the cobra is best suited to be an ally in a wave scenario.

[mirror176]

Was going to be my next single AI test. hover ai (or was it turtle ai) and semper fi also placed well on scoreboard of loaded AIs though that can be impacted by damage waves do to a single player and I don't know how scoring is really calculated (and not the only one, negative scores throw people's expectations off).

[mirror176]

Some current comparisons using the quick 'tutorial' glance noting the fps when the word 'base' is said on Win10 x64, 32GB RAM, i7-3820, NVIDIA GTX660 driver 474.44 (game crashes on driver update if open during process). Settings I didn't change: vertical sync=off, music was active. Other settings that I didn't change aren't listed in the options menus like displaying sensor radius=off (big impact in games). Some games are impacted by tabbing in/out of the program which I admit was being done some of the time (and no, I didn't test that impact+settings) as I was taking notes on the same 1 monitor machine. Some settings take effect on program restart; I tried to give a fair fresh launch to each with just reviewing menu to see settings stayed before launching tutorial. Some settings as combinations may have more of an impact. I suspect other issues like 'screen shake' being laggy may be resolved with changes like switching off OpenGL but did not yet test. Settings like Graphics=fullscreen vs desktop/windowed usually impact latency instead of fps in games; I did not do any latency tests here. I didn't test toggling the music manager entries to all be off as that would be a lot of clicking and I didn't know if it would matter in the end. I didn't test moving all audio sliders to 0 as there is no visual # and I don't know if it stops audio based logic or performs them still when they are inaudible. Listed % changes relative to my starting configuration; small changes may be within margin of error.

My default config was the first config I list. The settings I list after is just adjusted settings from that state. Game config under 4.3.5 as installed version with sequences.wz: shadow=on, fog=on, LOD distance=high, Graphics=Fullscreen, 1920x1080, texture=2048, anti-aliasing=4x, scale=150%, backend=OpenGL, HRTF=enabled, colored cursor=on, cursor size=100%, fps=733 shadow=off, fps=786 7.23% fog=off, fps=732 -.14% LOD distance=default, fps=746 1.77% Graphics=Desktop Full, fps=733 0% 640x480, scale=%100, fps=1555 112.14% 1280x720, fps=924 26.06% (large fluctuations of +300 around that 'word' timing so may be meaningless) texture=128, fps=741 1.09% anti-aliasing=off, fps=1097 49.66% anti-aliasing=2x, fps=908 23.87% anti-aliasing=16x, fps=229 -68.76% anti-aliasing=32x, fps=709 -3.27% (no crash and ran twice with approx same results) display scale=100%, fps=756 3.14% display scale=200%, fps=712 -2.86% OpenGL ES, fps=571 -22.1% Vulkan, fps=572 -21.96% DirectX (ANGLE), fps=646 -11.87% HRTF=off, fps=735 .27% colored cursor=off, fps=733 0% cursor size=200%, fps=733 0%

Config for version 58a4dd6 from September 14th warzone2100_win_x64_portable..zip and no sequences.wz: terrain quality=high, shadows=on, shadow res=normal(2048), shadow filtering=high, LOD distance=high, fog=on, groups menu=on, Graphics=Fullscreen, 1920x1080, texture=2048, anti-aliasing=4x, scale=150%, backend=OpenGL, HRTF=enabled, colored cursor=on, cursor size=100%, fps=214 (new look definitely coming at a cost on my machine, or its a less optimized prerelease) changes done in the same order with different settings at end. shadows=off, fps=214 0%(dropped by 10 until program relaunch) fog=off, fps=212 -.93% LOD distance=default, fps=295 37.85% Graphics=Desktop Full, fps=213 -.47%% 640x480, scale=%100, fps=912 326.17% 1280x720, fps=425 98.60% texture=128, fps=284 32.71% anti-aliasing=off, fps=259 21.03% anti-aliasing=2x, fps=228 6.54% anti-aliasing=16x, fps=81 -62.15% anti-aliasing=32x, fps=N/A (setting removed, guess someone coded that something was up. settings page still warns of anti-aliasing crashes and especially for >16) display scale=100%, fps=225 5.14% display scale=200%, fps=206 -3.74% OpenGL ES, fps=191 -10.75% Vulkan, fps=N/A - crash dialog box that I reported as #3383 . Like a good internet reviewer, I did not reboot my machine between tests, including when errors occurred...and we're testing . . . OpenGL, anti-aliasing=off, fps=258 20.56% Vulkan, anti-aliasing=off, fps=337 57.48% (oh, Vulkan can win on my hardware after all?) DirectX (ANGLE), fps=186 -13.08% HRTF=off, fps=213 -.47% colored cursor=off, fps=212 -.93% cursor size=200%, fps=214 0%

The new settings: terrain quality=classic, fps=537 150.93% terrain quality=normal, fps=423 97.66% (seems newer graphics are having fps impact) shadow res=high(4096), fps=208 -2.8% shadow filtering=low, fps=205 -4.21% shadow filtering=ultra, fps=216 .93% groups menu=off, fps=213 -.47%

Menu doesn't describe shadow settings rely on shadows=on; testing: shadows=off, shadow res=high(4096), fps=201 -6.07% shadows=off, shadow filtering=low, fps=205 -4.21% shadows=off, shadow filtering=ultra, fps=203 -5.14% interesting; the 2 drops in fps with shadows enabled vs disabled seem repeatable.

If someone wants any other configurations checked let me know. If your hardware or settings varies, then you may get different results and ratios of change.

Other observed interactions and suggestions while doing this testing (I will open separate issues if desired): LOD distance choice of 'default' should be relabeled if it is not 'default' anymore unless the portable copy read my nonportable settings. Music manager clicks check boxes on mousebutton down and uncheck on mousebutton release though it does only 1 action per down+up combination; should be consistent on reacting to the up (likely) or down (less likely).

Please consider making display scale be in center of screen or make mouse follow its position on the interface as it moves; not as important with resolution as it uses a (more appropriate? they are always slower unless going 2+ choices away) pulldown menu. It would be nice to have options to toggle message dismissal in their related settings menus (campaign video sequences missing, display scale being small, etc.) and menu options should be reconsidered for what is in graphics vs in video. Settings don't help user realize how it may impact hardware performance like loading CPU for shadows. Games usually mention when setting changes are now pending a restart instead of just saying it will be needed for the relevant settings but 'normally' should not matter unless doing something like me where you keep changing a lot + have a bad memory + need to know results are consistent.

I still have more to see but at a glance I definitely like the upcoming changes.