WIP: JSON output with `--write` flag

Aloso commented 10 months ago

Closes #10

When running cargo bench -- --write, produce a JSON file of the following shape:

{
  "main.parse.strings": {
    "display_name": "strings",
    "sample_count": 100,
    "iter_count": 200,
    "time": {
      "fastest": 921251,
      "slowest": 1322751,
      "median": 954001,
      "mean": 980421
    },
    "counts": {
      "bytes": {
        "fastest": 60,
        "slowest": 60,
        "median": 60,
        "mean": 60
      }
    }
  },
  ...
}

for this benchmark:

main               fastest       │ slowest       │ median        │ mean          │ samples │ iters
├─ parse                         │               │               │               │         │
│  ╰─ strings      921.2 ns      │ 1.322 µs      │ 954 ns        │ 980.4 ns      │ 100     │ 200
│                  65.12 MB/s    │ 45.36 MB/s    │ 62.89 MB/s    │ 61.19 MB/s    │         │
...

The JSON file is written to $CARGO_MANIFEST_PATH/target/divan/{time}.json, where {time} is a RFC3339 formatted timestamp, e.g. 2023-11-12T22:12:00.231816795Z.

I'm planning to create a web app to plot and compare different benchmarks, show changes over time, etc.

TODO:

[ ] create a nested JSON object, instead of joining benchmark names with .
[ ] make it an optional feature? (so users need divan = { version = "*", features = ["json"] } to opt-in)
[ ] include info about the git revision (branch, tag, commit, commit date)?

Feedback is appreciated!

polarathene commented 10 months ago

Just to keep in mind for review, the feature is ideally implemented in a way that other output formats can be supported too. There's a request for Markdown table output too which could potentially build off the work here.

With JSON output from this PR, other formats like CSV/TSV can technically be constructed from the JSON output via tools like jaq or yq (also can convert to YAML/TOML).

I'm not sure how useful additional outputs would be beyond JSON and Markdown which should be sufficient for most use-cases, with other output formats being mostly additional convenience?

OliverKillane commented 8 months ago

Any unpushed progress on this, given it has been draft for 65 days?

I also need json output for a report, so am considering just doing it myself.

nvzqz / divan

WIP: JSON output with `--write` flag #29