[ benchmark ] unable to use the json reporter

mouhannad-sh commented 4 months ago

Describe the bug

Hi 👋🏻
Thank you for adding the ability to run benchmarking inside of Vitest, it's 🔥

In the documentation for benchmark outputfile the json reporter was mentioned as the only way to output benchmark results and compare against previous results via the compare option However, when I follow the instructions to configure my vitest benchmarks I get this error

Error: Failed to load custom Reporter from json

Additionally, I noticed that the BenchmarkBuiltinReporters only include "default" and "verbose" and both don't seem to allow us to output benchmark results into a file.

I would like to be able to export the benchmark results and compare them to a previous run if possible

Reproduction

run npm run bench

https://stackblitz.com/edit/vitest-dev-vitest-1qwwb1?file=vite.config.ts

System Info

System:
    OS: Linux 5.0 undefined
    CPU: (8) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 0 Bytes / 0 Bytes
    Shell: 1.0 - /bin/jsh
  Binaries:
    Node: 18.20.3 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 10.2.3 - /usr/local/bin/npm
    pnpm: 8.15.6 - /usr/local/bin/pnpm
  npmPackages:
    @vitest/ui: latest => 1.6.0 
    vite: latest => 5.3.1 
    vitest: latest => 1.6.0

Used Package Manager

npm

Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

hi-ogawa commented 4 months ago

It looks like the documentation of benchmark.outputFile is a little outdated. For comparison, you need to use benchmark.outputJson and benchmark.compare combo as shown in https://vitest.dev/config/#benchmark-outputJson

# save main branch's result
git checkout main
vitest bench --outputJson main.json

# change a branch and compare against main
git checkout feature
vitest bench --compare main.json

Technically, you can put a same file to both options like below, but this will always overwrite bench.json, so that's a bit different from the intended usage.

export default defineConfig({
  test: {
    benchmark: {
      outputJson: './bench.json',
      compare: './bench.json'
    },
  },
});

mouhannad-sh commented 4 months ago

Thanks @hi-ogawa this works with a small gotcha

It will fail at the first runtime if there's no bench.json file present. might be worth adding this detail to the docs too

I updated the Stackblitz link to reflect this thanks a bunch !

vitest-dev / vitest