chmoder / criterion.dev

The official support channel for https://api.criterion.dev and https://criterion.dev
https://criterion.dev
6 stars 0 forks source link

Update guide to handle multiple benchmarks #3

Closed str4d closed 3 years ago

str4d commented 3 years ago

In some situations, criterion generates one raw.csv file per benchmark (for example, when setting the parameter argument to BenchmarkId). The current usage instructions only work for a single raw.csv file. It would be helpful to amend the instructions to support multiple files (I presume this would just require looping over the discovered files and uploading each one individually, but I don't know the specifics of what the server expects).

chmoder commented 3 years ago

Hmm... good point! Adding that feature should be pretty straight forward. If you have some time would you mind adding some extra information?

I'm trying to discern if the file is generated because of the Cargo.toml, benchmark file in dir: benches, or the group, or possibly something else.

If you have a project I can clone, just a link to that would be enough. Otherwise, things that would help me add the feature would be:

chmoder commented 3 years ago

@str4d, I looking into how criterion may create multiple raw csv files. There are at least two ways to do it, and they both end up using the same file structure. So I will be working on adding support for this in the next week. Thanks for the suggestion - https://github.com/chmoder/data_vault/pull/4 is the example I modified to get multiple methods benchmarked at once.

chmoder commented 3 years ago

The bash uploader is now more generic! This means:

1) It does not run criterion or any benchmark software automatically 2) It supports uploading multiple benchmarks at once 3) It relies on the default criterion behavior where baselines are saved in a directory called new 4) A simplified and more expected usage 5) Can adopt other benchmark frameworks

Please see the examples on https://criterion.dev/ for the changes