Criterion version 0.4 and 0.5 milestones

[lemmih]

What should be in criterion-0.4 and criterion-0.5? Few things are set in stone so submit your comments and ideas.

Version 0.4:

• [x] Remove all deprecated functions.
• [x] Make CSV output an optional and non-default feature.
• [x] Make HTML output an optional and non-default feature.
• [x] Make rayon an optional and default feature. Rayon improves the analysis performance but isn't required for correctness.
• [x] Make plotters an optional and non-default feature.
• [x] Print status messages to stderr, benchmark results to stdout.
• [x] Allow durations from command-line arguments to be fractions of a second. For example, --warm-up-time=0.5.
• [x] Formally support WASI.

Version 0.5:

• [ ] Remove CSV output feature. Replaced by JSON.
• [ ] Remove HTML output feature. HTML plots will be generated by cargo-criterion.

Random wishlist

• Unify Criterion and BenchmarkGroup such that there aren't two versions of sample_size, warm_up_time, etc.
• Programmatic interface to benchmark results and baselines.
• Better way of printing outliers, see #292.
• Display both time and throughput, see #382.
• Better alignment of output in the bencher format.
• Consistent units for grouped benchmarks, see #367.

[bheisler]

My intention was actually to move away from relying on Gnuplot and make Plotters the default plotting backend; that way users don't need to install Gnuplot separately, but that's not set in stone. Is there some reason we want to avoid using Plotters?

[c410-f3r]

External installation of the gnuplot lib has always been a nuisance. Let's say that with plotters everything "just works".

[poliorcetics]

What's missing for a release of v0.4 ? I see all the TODOs are checked

[lemmih]

The code is there. Just needs to be reviewed.

[dclong]

Can we use a JS library for plots so that they are more interactive?

https://github.com/bheisler/criterion.rs/issues/653

[wilgaboury]

I can understand CSV being an optional feature, but what is the reasoning behind it being removed? It is extremely useful for seamlessly importing and doing data visualization in other languages or programs (R, Python, Excel...).