We need to switch to criterion for the following reasons:
Criterion has warm-up option which is really important for running with Wasmer since we want to disregard the time it takes for it to warm-up;
Criterion uses multiple iterations (it runs the same command multiple times and then divides the time spent by the number of runs). It is very helpful in discarding further warm-up effects;
It can compare its execution result to the previous execution and tell when the change is statistically significant based on p-values, which is useful for decision making.
It can be used to compare code that uses different configurations (e.g. disabled/enabled gas).
We need to know when our changes cause smart contract to run slower or faster.
Currently we have some benchmarks but they use bencher, which is problematic because it is too simplistic: https://github.com/nearprotocol/nearcore/blob/deee97ac4c29d61095770d04373bc5619a490926/runtime/near-vm-runner/benches/bench.rs
We need to switch to criterion for the following reasons:
We can follow the following code to start using criterion: https://github.com/nearprotocol/borsh/blob/master/borsh-rs/benchmarks/benches/bench.rs