hohav
commented
1 year ago This is a good idea, but I lack experience with Rust benchmarking so I can't offer much advice there.
The data set is another interesting question. A wide variety of games is definitely a good idea, but do we need to go "deep" too for robustness? I.e. do we need a largeish set of replays, or is a small number of handpicked samples sufficient as long as they're diverse enough?
I've been looking into adding benchmarks. It would also help clarify if certain changes have a justifiable performance impact (#24, #25). Right now, I'm looking at Criterion (https://github.com/bheisler/cargo-criterion). We may also want to use Iai (https://github.com/bheisler/iai) for CI benchmarks.
If anyone has experience with either of these tools (or something else) I would appreciate some guidance.
Also, we need some set of replay files that cover different performance scenarios. Different matchups, Ice Climbers games, long games, short games, netplay games, local games, non-standard games (ie. not 4-stock 8 minutes), etc. I wonder if it would make sense to store the replay set compressed w/ peppi and inflate each time we run the benchmarks.