ordo-one / package-benchmark

Swift benchmark runner with many performance metrics and great CI support
Apache License 2.0
326 stars 25 forks source link

fix(minor): Fail CI also for update benchmark baseline updates that fails due to EPERM #212

Closed hassila closed 11 months ago

hassila commented 11 months ago

Description

To ensure we don't "succeed" when the benchmark throws due to permissioning errors (e.g. networking without disabling sandbox)

How Has This Been Tested?

Manually tested with distributed system.

Minimal checklist:

codecov[bot] commented 11 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (645e475) 70.40% compared to head (6b00da3) 70.40%.

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/212/graphs/tree.svg?width=650&height=150&src=pr&token=hXHmhEG1iF&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one)](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/212?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) ```diff @@ Coverage Diff @@ ## main #212 +/- ## ======================================= Coverage 70.40% 70.40% ======================================= Files 29 29 Lines 3909 3909 ======================================= Hits 2752 2752 Misses 1157 1157 ``` ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/212?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/212?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one). Last update [645e475...6b00da3](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/212?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one).
github-actions[bot] commented 11 months ago

Pull request benchmark comparison [ubuntu-latest] with 'main' run at 2023-12-20T16:08:58+00:00 Pull request had performance regressions