ordo-one / package-benchmark

Swift benchmark runner with many performance metrics and great CI support
Apache License 2.0
326 stars 25 forks source link

feat(minor): Add support for metricP90AbsoluteThresholds export format #180

Closed hassila closed 1 year ago

hassila commented 1 year ago

Description

This provides better out-of-the-box experience for absolute checks and is geared towards CI checking of e.g. malloc/syscall deviations. The thresholds are expected to be in a "Thresholds" directory.

To generate a new baseline, run e.g.

swift package --allow-writing-to-package-directory benchmark --format metricP90AbsoluteThresholds --path Thresholds/

To run a check vs. the saved thresholds, run:

swift package benchmark baseline check --check-absolute-path /relative/or/absolute/path/to/Thresholds

See the P90AbsoluteThresholds sample benchmark target for typical usage.

How Has This Been Tested?

Manual testing, added additional benchmark target.

Minimal checklist:

codecov[bot] commented 1 year ago

Codecov Report

Merging #180 (837cdb2) into main (6e0a077) will not change coverage. The diff coverage is 100.00%.

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/180/graphs/tree.svg?width=650&height=150&src=pr&token=hXHmhEG1iF&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one)](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/180?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) ```diff @@ Coverage Diff @@ ## main #180 +/- ## ======================================= Coverage 71.29% 71.29% ======================================= Files 27 27 Lines 3501 3501 ======================================= Hits 2496 2496 Misses 1005 1005 ``` | [Files Changed](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/180?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) | Coverage Δ | | |---|---|---| | [Tests/BenchmarkTests/AdditionalTests.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/180?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-VGVzdHMvQmVuY2htYXJrVGVzdHMvQWRkaXRpb25hbFRlc3RzLnN3aWZ0) | `100.00% <100.00%> (ø)` | | | [Files Changed](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/180?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) | Coverage Δ | | |---|---|---| | [Tests/BenchmarkTests/AdditionalTests.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/180?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-VGVzdHMvQmVuY2htYXJrVGVzdHMvQWRkaXRpb25hbFRlc3RzLnN3aWZ0) | `100.00% <100.00%> (ø)` | | ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/180?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/180?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one). Last update [6e0a077...837cdb2](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/180?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one).
github-actions[bot] commented 1 year ago

Pull request benchmark comparison [ubuntu-latest] with 'main' run at 2023-08-23T15:25:05+00:00 Pull request had performance regressions