ordo-one / package-benchmark

Swift benchmark runner with many performance metrics and great CI support
Apache License 2.0
326 stars 25 forks source link

feat: Errors thrown no longer fail entire benchmark process #166

Closed ORyanHampton closed 1 year ago

ORyanHampton commented 1 year ago

Description

When running benchmarks, if even one failed, we would lose all collected data. These changes output collected benchmarks and drop the data from failed benchmarks. This helps prevent data loss on benchmarks that aren't failing.

How Has This Been Tested?

Ran locally and checked output

Minimal checklist:

codecov[bot] commented 1 year ago

Codecov Report

Merging #166 (ea88579) into main (9cc05dc) will not change coverage. The diff coverage is n/a.

:exclamation: Current head ea88579 differs from pull request most recent head f71da60. Consider uploading reports for the commit f71da60 to get more accurate results

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/166/graphs/tree.svg?width=650&height=150&src=pr&token=hXHmhEG1iF&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one)](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/166?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) ```diff @@ Coverage Diff @@ ## main #166 +/- ## ======================================= Coverage 71.42% 71.42% ======================================= Files 26 26 Lines 3446 3446 ======================================= Hits 2461 2461 Misses 985 985 ``` [see 1 file with indirect coverage changes](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/166/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) [see 1 file with indirect coverage changes](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/166/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/166?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/166?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one). Last update [d3b66be...f71da60](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/166?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one).