Closed NickCondron closed 7 months ago
This is a good idea, but I lack experience with Rust benchmarking so I can't offer much advice there.
The data set is another interesting question. A wide variety of games is definitely a good idea, but do we need to go "deep" too for robustness? I.e. do we need a largeish set of replays, or is a small number of handpicked samples sufficient as long as they're diverse enough?
Implemented via #43 (b62b81156f7db999239c49a2c0c886338d5b3bac and 0c671c82ffcc633b3540dac518ef737ed7e7c6b4).
I've been looking into adding benchmarks. It would also help clarify if certain changes have a justifiable performance impact (#24, #25). Right now, I'm looking at Criterion (https://github.com/bheisler/cargo-criterion). We may also want to use Iai (https://github.com/bheisler/iai) for CI benchmarks.
If anyone has experience with either of these tools (or something else) I would appreciate some guidance.
Also, we need some set of replay files that cover different performance scenarios. Different matchups, Ice Climbers games, long games, short games, netplay games, local games, non-standard games (ie. not 4-stock 8 minutes), etc. I wonder if it would make sense to store the replay set compressed w/ peppi and inflate each time we run the benchmarks.