search

inikep / lzbench

lzbench is an in-memory benchmark of open-source LZ77/LZSS/LZMA compressors
883 stars 180 forks source link

Timings ? #47

Open g1mv opened 6 years ago

g1mv commented 6 years ago

Are timings done by calculating the actual process time or an absolute time ? I'm just wondering as a quick look at the code made me think it might be the latter case. That could be problematic as a heavily or variably loaded test machine would generate distorted results.

inikep commented 6 years ago

Time is measured with: QueryPerformanceCounter for Windows mach_absolute_time for macOS clock_gettime(CLOCK_MONOTONIC) for Linux

g1mv commented 6 years ago

This might be an issue then, because the timings cannot be reliable if anything else runs on the testing machine, as these functions do not count CPU times but absolute times. I actually encountered the problem a while ago and made a simple lib (cputime) to access multiplatform CPU times. Basically, on Linux and MacOS, you can use getrusage() and check the ru_utime value; and on Windows, GetProcessTimes() will give user time values as well. These functions are not microsecond accurate and are certainly less accurate than the ones you're using, but they have a big advantage which is not being dependent on the external load of your testing machine, i.e. should be immune to load variations and give consistent results. That's what should be expected from a benchmark application I suppose. What do you think ?

inikep commented 6 years ago

Currently it works well (for lzbench and TurboBench) because we set realtime priority and assume that we are using only a single core. Glza is only compressor that uses multi-core but it's slow anyway. The task of comparing compression on multi-core is much harder.