bencherdev / bencher

🐰 Bencher - Continuous Benchmarking
https://bencher.dev
Other
505 stars 22 forks source link

js_benchmark adapter not working as expected #506

Open arv opened 1 day ago

arv commented 1 day ago

I'm trying to use benchmark data outputted in a Benchmark.js format but the adapter fails with "Message: Failed to convert results with adapter". Here is my CLI output:

bencher run --project test-repl --adapter js_benchmark "node src/runner.js --groups hash"
(node:96948) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Running 6 benchmarks on Chromium...
text encoder utf8 x 28.33 ops/sec ±3.3% (13 runs sampled)
text encoder utf16 x 55.25 ops/sec ±16.6% (19 runs sampled)
sha512 wasm from string utf8 x 15.20 ops/sec ±8.8% (7 runs sampled)
sha512 wasm from string utf16 x 19.23 ops/sec ±1.7% (8 runs sampled)
sha512 native from string utf8 x 21.46 ops/sec ±4.1% (9 runs sampled)
sha512 native from string utf16 x 29.76 ops/sec ±6.8% (14 runs sampled)
Done!

Benchmark Harness Results:
Running 6 benchmarks on Chromium...
text encoder utf8 x 28.33 ops/sec ±3.3% (13 runs sampled)
text encoder utf16 x 55.25 ops/sec ±16.6% (19 runs sampled)
sha512 wasm from string utf8 x 15.20 ops/sec ±8.8% (7 runs sampled)
sha512 wasm from string utf16 x 19.23 ops/sec ±1.7% (8 runs sampled)
sha512 native from string utf8 x 21.46 ops/sec ±4.1% (9 runs sampled)
sha512 native from string utf16 x 29.76 ops/sec ±6.8% (14 runs sampled)
Done!

Bencher New Report:
{
  "branch": "main",
  "end_time": "2024-09-19T13:08:38.275254Z",
  "hash": "0dde655f9dadea845a67fef1839d56bda0331c19",
  "results": [
    "Running 6 benchmarks on Chromium...\ntext encoder utf8 x 28.33 ops/sec ±3.3% (13 runs sampled)\ntext encoder utf16 x 55.25 ops/sec ±16.6% (19 runs sampled)\nsha512 wasm from string utf8 x 15.20 ops/sec ±8.8% (7 runs sampled)\nsha512 wasm from string utf16 x 19.23 ops/sec ±1.7% (8 runs sampled)\nsha512 native from string utf8 x 21.46 ops/sec ±4.1% (9 runs sampled)\nsha512 native from string utf16 x 29.76 ops/sec ±6.8% (14 runs sampled)\nDone!"
  ],
  "settings": {
    "adapter": "js_benchmark"
  },
  "start_time": "2024-09-19T13:08:32.254104Z",
  "testbed": "localhost"
}

Failed to create new report: Error processing request:
Status: 400 Bad Request
Headers: {"content-type": "application/json", "x-request-id": "7d9999dd-4420-44cd-b2f1-f4fdeb5b5af2", "content-length": "687", "date": "Thu, 19 Sep 2024 13:08:38 GMT", "server": "Fly/a5e9a2096 (2024-09-17)", "via": "2 fly.io", "fly-request-id": "01J856C8GGM8BC930BXS60HWVF-cdg"}
Request ID: 7d9999dd-4420-44cd-b2f1-f4fdeb5b5af2
Message: Failed to convert results with adapter (js_benchmark | JsonReportSettings { adapter: Some(JsBenchmark), average: None, fold: None }): Failed to convert results: Running 6 benchmarks on Chromium...
text encoder utf8 x 28.33 ops/sec ±3.3% (13 runs sampled)
text encoder utf16 x 55.25 ops/sec ±16.6% (19 runs sampled)
sha512 wasm from string utf8 x 15.20 ops/sec ±8.8% (7 runs sampled)
sha512 wasm from string utf16 x 19.23 ops/sec ±1.7% (8 runs sampled)
sha512 native from string utf8 x 21.46 ops/sec ±4.1% (9 runs sampled)
sha512 native from string utf16 x 29.76 ops/sec ±6.8% (14 runs sampled)
Done!

I also tried after removing the non benchmark.js lines with the same result.

bencher run --project test-repl --adapter js_benchmark "node --no-warnings src/runner.js --groups hash"
text encoder utf8 x 28.01 ops/sec ±1.9% (13 runs sampled)
text encoder utf16 x 55.87 ops/sec ±5.7% (19 runs sampled)
sha512 wasm from string utf8 x 16.00 ops/sec ±7.2% (7 runs sampled)
sha512 wasm from string utf16 x 19.12 ops/sec ±1.6% (8 runs sampled)
sha512 native from string utf8 x 21.05 ops/sec ±4.7% (9 runs sampled)
sha512 native from string utf16 x 29.67 ops/sec ±2.7% (14 runs sampled)

Benchmark Harness Results:
text encoder utf8 x 28.01 ops/sec ±1.9% (13 runs sampled)
text encoder utf16 x 55.87 ops/sec ±5.7% (19 runs sampled)
sha512 wasm from string utf8 x 16.00 ops/sec ±7.2% (7 runs sampled)
sha512 wasm from string utf16 x 19.12 ops/sec ±1.6% (8 runs sampled)
sha512 native from string utf8 x 21.05 ops/sec ±4.7% (9 runs sampled)
sha512 native from string utf16 x 29.67 ops/sec ±2.7% (14 runs sampled)

Bencher New Report:
{
  "branch": "main",
  "end_time": "2024-09-19T13:18:50.835927Z",
  "hash": "0dde655f9dadea845a67fef1839d56bda0331c19",
  "results": [
    "text encoder utf8 x 28.01 ops/sec ±1.9% (13 runs sampled)\ntext encoder utf16 x 55.87 ops/sec ±5.7% (19 runs sampled)\nsha512 wasm from string utf8 x 16.00 ops/sec ±7.2% (7 runs sampled)\nsha512 wasm from string utf16 x 19.12 ops/sec ±1.6% (8 runs sampled)\nsha512 native from string utf8 x 21.05 ops/sec ±4.7% (9 runs sampled)\nsha512 native from string utf16 x 29.67 ops/sec ±2.7% (14 runs sampled)"
  ],
  "settings": {
    "adapter": "js_benchmark"
  },
  "start_time": "2024-09-19T13:18:44.817001Z",
  "testbed": "localhost"
}

Failed to create new report: Error processing request:
Status: 400 Bad Request
Headers: {"content-type": "application/json", "x-request-id": "da9d022d-f7a9-4b54-a1e4-61c81f5a4c21", "content-length": "642", "date": "Thu, 19 Sep 2024 13:18:51 GMT", "server": "Fly/a5e9a2096 (2024-09-17)", "via": "2 fly.io", "fly-request-id": "01J856YYM2H8BS5R4JYJM3437K-cdg"}
Request ID: da9d022d-f7a9-4b54-a1e4-61c81f5a4c21
Message: Failed to convert results with adapter (js_benchmark | JsonReportSettings { adapter: Some(JsBenchmark), average: None, fold: None }): Failed to convert results: text encoder utf8 x 28.01 ops/sec ±1.9% (13 runs sampled)
text encoder utf16 x 55.87 ops/sec ±5.7% (19 runs sampled)
sha512 wasm from string utf8 x 16.00 ops/sec ±7.2% (7 runs sampled)
sha512 wasm from string utf16 x 19.12 ops/sec ±1.6% (8 runs sampled)
sha512 native from string utf8 x 21.05 ops/sec ±4.7% (9 runs sampled)
sha512 native from string utf16 x 29.67 ops/sec ±2.7% (14 runs sampled)
epompeii commented 5 hours ago

@arv thank you for the bug report, and I'm sorry you're running into trouble. This is a bug in the js_benchmark adapter: https://github.com/bencherdev/bencher/blob/main/lib/bencher_adapter/src/adapters/js/benchmark.rs

Your original benchmark harness results will be turned into a test case to make sure that this is resolved. If you have any interest, I'm open to a PR that fixes the bug. Otherwise, I'll try and get to this soon.