haskell-github-trust / thyme

A faster date and time library based on time
BSD 3-Clause "New" or "Revised" License
46 stars 33 forks source link

Implement time parsing benchmarks #67

Closed parsonsmatt closed 5 years ago

parsonsmatt commented 6 years ago

This PR implements some benchmarks comparing various time parsing functionality. The difference is pretty significant:

benchmarking parsing/Thyme.parseTime
time                 2.042 μs   (2.004 μs .. 2.080 μs)
                     0.998 R²   (0.996 R² .. 1.000 R²)
mean                 1.989 μs   (1.973 μs .. 2.018 μs)
std dev              73.99 ns   (49.60 ns .. 122.6 ns)
variance introduced by outliers: 50% (severely inflated)

benchmarking parsing/Time.parseTimeM
time                 3.253 μs   (3.240 μs .. 3.269 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 3.253 μs   (3.229 μs .. 3.304 μs)
std dev              112.9 ns   (60.74 ns .. 221.4 ns)
variance introduced by outliers: 46% (moderately inflated)

benchmarking parsing/Thyme.timeParser
time                 1.334 μs   (1.328 μs .. 1.342 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.333 μs   (1.327 μs .. 1.341 μs)
std dev              23.75 ns   (17.22 ns .. 34.88 ns)
variance introduced by outliers: 19% (moderately inflated)
erikd commented 6 years ago

variance introduced by outliers: 50% (severely inflated)

I'm a little hesitant to trust benchmarks where the outliers are so high. A couple of things I have found to help:

parsonsmatt commented 6 years ago

Indeed! The benchmarks were broken.

Benchmark bench: RUNNING...
benchmarking parsing/Thyme.parseTime
time                 1.586 μs   (1.580 μs .. 1.593 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 1.571 μs   (1.564 μs .. 1.578 μs)
std dev              23.41 ns   (19.75 ns .. 27.87 ns)
variance introduced by outliers: 14% (moderately inflated)

benchmarking parsing/Time.parseTimeM
time                 9.358 μs   (9.232 μs .. 9.502 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 9.209 μs   (9.169 μs .. 9.289 μs)
std dev              183.6 ns   (124.9 ns .. 295.6 ns)
variance introduced by outliers: 19% (moderately inflated)

benchmarking parsing/Thyme.timeParser
time                 1.072 μs   (1.066 μs .. 1.078 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 1.066 μs   (1.060 μs .. 1.075 μs)
std dev              24.68 ns   (16.43 ns .. 40.82 ns)
variance introduced by outliers: 29% (moderately inflated)

Benchmark bench: FINISH