triton-lang / triton

Development repository for the Triton language and compiler
https://triton-lang.org/
MIT License
13.15k stars 1.61k forks source link

why the benchmark uses median data instead of avg data? #3754

Open sleepwalker2017 opened 6 months ago

sleepwalker2017 commented 6 months ago
   quantiles = [0.5, 0.2, 0.8]
    if provider == 'cublas':
        ms, min_ms, max_ms = triton.testing.do_bench(lambda: torch.matmul(a, b), quantiles=None)

I see the code here, it uses the median time cost instead of the avg data.

In my benchmark, the result for cublas varies a lot in different test. I don't know why.

yunjiangster commented 4 months ago

Median is more stable than avg in statistics. It reduces outlier impact.