search

ordo-one / package-benchmark

Swift benchmark runner with many performance metrics and great CI support
Apache License 2.0
326 stars 25 forks source link

The benchmark results in an error but the `check` shows success #289

Open MahdiBM opened 1 month ago

MahdiBM commented 1 month ago 
swift package --disable-sandbox \
      benchmark thresholds check \
      --path $PWD/Benchmarks/Thresholds/ \
      --format markdown

succeeds when the actual benchmark has resulted in an error.

hassila commented 1 month ago

You mean if the benchmark crashed, or?

MahdiBM commented 1 month ago

No the benchmark run doesn't crash. It simply throws an error.

I noticed the benchmark says Checking ["benchmark name 1", "benchmark name 2"] around the end, but if the benchmark run has resulted in an error, it doesn't mention that benchmark's name in there at all.

Probably all benchmark runs for a specific benchmark must fail for this to happen. Then the Benchmarking package doesn't have any results for that certain benchmark to compare to, so it simply skips the comparison instead of complaining that where are the results that I'm supposed to compare to or something. It should just fail the benchmark comparison if any of the benchmarks resulted in an error.