bencheeorg / benchee

Easy and extensible benchmarking in Elixir providing you with lots of statistics!
MIT License
1.42k stars 66 forks source link

Comparison broken when saving and loading #287

Closed PragTob closed 5 years ago

PragTob commented 5 years ago
Comparison: 
flat_map                       2.41 K
map.flatten                    1.29 K - 1.87x slower +362.05 μs
flat_map (first-try)           2.39 K map.flatten (first-try)        1.26 K - 1.91x slower +378.52 μs

Extended statistics: 

Name                            minimum        maximum    sample size                     mode
flat_map                      345.18 μs     1195.24 μs        12.00 K     405.28 μs, 405.28 μs
map.flatten                   494.08 μs     1479.42 μs         6.42 K                739.22 μs
flat_map (first-try)          345.21 μs     1273.69 μs        11.94 K     405.52 μs, 405.46 μs
map.flatten (first-try)       493.44 μs     1614.85 μs         6.27 K                741.30 μs

See our sample for that.

I'm guessing it is because we didn't compute relative statistics for it... which is true because they are loaded after statistics calculation. This means we probably should have a separate Relative/Meta Statistics step that is called after loading them and always recomputed.

That probably has other problems again (such as sorting etc...) which I'll then have to figure out tomorrow.