search

adnanaziz / EPIJudge

EPI Judge - Preview Release
Other
2.83k stars 1.88k forks source link

Bug: wildly fluctuating test times #202

Open sandeep-datta opened 3 years ago

sandeep-datta commented 3 years ago

Edit: I just realized I may have been looking at the wrong metric. What does the 2 us in Test PASSED (10001/10001) [ 2 us] stand for?

General issue Classification: Buggy performance testing code. Reproducible test case: ./repro # https://gist.github.com/sandeep-datta/f9706238e7a65c0d6aad545fde533aae Filtered output: https://gist.github.com/sandeep-datta/e3f956fda7210a401fbc70ab3dddd0b3 Filter applied to saved output: grep Test output | grep us | sort -k 2

Suggested fix: run the test 100 times (for any test that takes less than 1 ms to run), remove outliers and show the average of collected run times.

4x3lpri3t0 commented 3 years ago

I believe that 2 us stands for 2 microseconds (0.002 miliseconds).

metopa commented 3 years ago

Hi @sandeep-datta , thank you for noticing that. Right now we're in the process of rewriting internal infrastructure. After that we'll improve the precision of measuring execution time.