search

adventuregamestudio / ags

AGS editor and engine source code
Other
695 stars 158 forks source link

benchmark mode #821

Open rofl0r opened 5 years ago

rofl0r commented 5 years ago

it's currently pretty hard to properly benchmark codechanges.

@sonneveld suggested the following in #815 :

A rough check might be to run the engine with --fps and use ctrl-e to run at max frame rate.

i tried this out, but the FPS jump quite a bit, so it is pretty unreliable.

i think a good way to measure performance would be a special mode (could be triggered with --benchmark), which, upon receipt of a certain signal (could be ctrl-e as well) runs at max framerate for a specified timeframe, e.g. 60 seconds (this could be an option to --benchmark switch), and then terminates the program and prints the average framerate for that time (the user-based triggering is needed so one can load a specific savegame and prepare things) optimally, the mode could even suppress video and sound output, so only the raw speed of the engine is measured, without artifial caps due to hardware driver limitations. instead of average fps, the total number of loop iterations which is currently shown in FPS display could be a good indicator for performance too.

morganwillcock commented 5 years ago

I think that potentially you may also need a forced 'max speed and full screen' update mode to prevent any kind of overlay that is updating more quickly than the game from creating a smearing effect on the screen e.g. Steam Acheivement pop-ups leaving a trail on the game display or on the black masking bars.

So perhaps this feature is for benchmarking but could also be relevant for scripts and plug-ins.

rofl0r commented 5 years ago

turns out there's a way the fps display can be used to reason about performance improvements. this is when a game is used that's extremely script-heavy, such as "Operation Forklift".

for example, when using my new assembly optimizer pass in agsutils, i get almost 20% higher FPS (without the optimization pass, the game doesn't even reach the targeted 40 FPS).

therefore, if we had some sort of specialized benchmark testgame, which does a lot of script stuff and very rarely updates the screen, this could be a halfways reliable way to benchmark. basically that testgame only needs a single loop that's repeated very often for every frame, and uses all possibilities of the VM. e.g. it needs to do function calls to external scripts, to the same script, read and write stuff to the memory (script objects), array indices, all arithmetic / boolean/ logical ops, etc. optimally, that game achieves max 40 fps on very fast desktop machines, so even small performance improvements in the order of 0.5% can be reliably measured.

sonneveld commented 5 years ago

therefore, if we had some sort of specialized benchmark testgame, which does a lot of script stuff and very rarely updates the screen, this could be a halfways reliable way to benchmark.

Sounds familiar :) https://github.com/adventuregamestudio/ags/pull/847#issuecomment-510300265

It would be useful if someone had the time to make this benchmark game though.

rofl0r commented 5 years ago

It would be useful if someone had the time to make this benchmark game though.

an alternative would be to take an existing game for which the sources are available, and simply put the program mainloop (before updating the screen) into a loop that's repeated like 1000x instead of once. maybe @ivan-mogilko's racing game ?