search

ordo-one / package-benchmark

Swift benchmark runner with many performance metrics and great CI support
Apache License 2.0
315 stars 25 forks source link

The benchmark results in an error but the `check` shows success #289

Open MahdiBM opened 1 week ago

MahdiBM commented 1 week ago 
swift package --disable-sandbox \
      benchmark thresholds check \
      --path $PWD/Benchmarks/Thresholds/ \
      --format markdown

succeeds when the actual benchmark has resulted in an error.

hassila commented 1 week ago

You mean if the benchmark crashed, or?

MahdiBM commented 1 week ago

No the benchmark run doesn't crash. It simply throws an error.

I noticed the benchmark says Checking ["benchmark name 1", "benchmark name 2"] around the end, but if the benchmark run has resulted in an error, it doesn't mention that benchmark's name in there at all.

Probably all benchmark runs for a specific benchmark must fail for this to happen. Then the Benchmarking package doesn't have any results for that certain benchmark to compare to, so it simply skips the comparison instead of complaining that where are the results that I'm supposed to compare to or something. It should just fail the benchmark comparison if any of the benchmarks resulted in an error.