search

Bodigrim / tasty-bench

Featherlight benchmark framework, drop-in replacement for criterion and gauge.
https://hackage.haskell.org/package/tasty-bench
MIT License
80 stars 11 forks source link

Some benchmarks written for `criterion` just complete immediately with `tasty-bench`. #54

Closed klapaucius closed 6 months ago

klapaucius commented 6 months ago 
All.Comparison.1000.find.vector-hashtables,6580352,647420
All.Comparison.1000.find.vector-hashtables (frozen),1469,38

the same benchmarks ran using criterion 

Comparison/1000/find/vector-hashtables,7.08775428580093e-6,6.878445436163731e-6,7.573894628958587e-6,1.0076997615131766e-6,5.435319381788577e-7,1.8001436015276278e-6
Comparison/1000/find/vector-hashtables (frozen),3.011253725428155e-6,2.997528898847814e-6,3.034834992200024e-6,5.824131186108332e-8,4.132973586165513e-8,8.391675029687467e-8

https://github.com/klapaucius/vector-hashtables/blob/2dc4c0dc2d9471bec8f1d1b8edd161fcc78250fc/bench/Main.hs#L52

 +RTS --info
 [("GHC RTS", "YES")
 ,("GHC version", "8.10.7")
 ,("RTS way", "rts_v")
 ,("Build platform", "x86_64-unknown-mingw32")
 ,("Build architecture", "x86_64")
 ,("Build OS", "mingw32")
 ,("Build vendor", "unknown")
 ,("Host platform", "x86_64-unknown-mingw32")
 ,("Host architecture", "x86_64")
 ,("Host OS", "mingw32")
 ,("Host vendor", "unknown")
 ,("Target platform", "x86_64-unknown-mingw32")
 ,("Target architecture", "x86_64")
 ,("Target OS", "mingw32")
 ,("Target vendor", "unknown")
 ,("Word size", "64")
 ,("Compiler unregisterised", "NO")
 ,("Tables next to code", "YES")
 ,("Flag -with-rtsopts", "")
 ]
Bodigrim commented 6 months ago

That's probably expected: https://github.com/Bodigrim/tasty-bench/blob/bb8810e3ca2d9752c4cf080f528eee3f96186074/src/Test/Tasty/Bench.hs#L1449-L1453 For some reason GHC optimizer does not get that aggressive with criterion harness, but that's a matter of luck. I think it would be nice to change 

-fvhfind :: Int -> VH.FrozenDictionary V.Vector Int V.Vector Int -> IO Int
-fvhfind n ht = return $ go 0 0 where
+fvhfind :: Int -> VH.FrozenDictionary V.Vector Int V.Vector Int -> Int
+fvhfind n ht = go 0 0 where

and then

-            , bench "vector-hashtables (frozen)" $ nfIO (fvhfind n fh) ]
+            , bench "vector-hashtables (frozen)" $ nf (fvhfind n) fh ]
klapaucius commented 6 months ago

Well, this works. Thanks.