feat(minor): Support thresholds for absolute checks

[hassila]

Description

For benchmarks that are not completely stable (in e.g. syscall/malloc count) due to use of e.g. async or networking, it is desirable to also be able to specify some leeway for benchmarks even for the absolute checks from thresholds.

This fixes https://github.com/ordo-one/package-benchmark/issues/220

By default the behaviour for absolute thresholds checks are as previously (strict, zero improvements/regressions are allowed), but now it is possible to define both absolute and relative thresholds for the benchmarks (as documented at https://swiftpackageindex.com/ordo-one/package-benchmark/1.21.3/documentation/benchmark/writingbenchmarks)

How Has This Been Tested?

Manual testing.

Minimal checklist:

• [x] I have performed a self-review of my own code
• [ ] I have added DocC code-level documentation for any public interfaces exported by the package
• [ ] I have added unit and/or integration tests that prove my fix is effective or that my feature works

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (02b4da9) 67.18% compared to head (1db45b5) 66.84%.

Additional details and impacted files

[](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) ```diff @@ Coverage Diff @@ ## main #223 +/- ## ========================================== - Coverage 67.18% 66.84% -0.35% ========================================== Files 34 33 -1 Lines 4376 4303 -73 ========================================== - Hits 2940 2876 -64 + Misses 1436 1427 -9 ``` | [Files](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) | Coverage Δ | | |---|---|---| | [Sources/Benchmark/BenchmarkClock.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvQmVuY2htYXJrQ2xvY2suc3dpZnQ=) | `33.72% <ø> (ø)` | | | [Sources/Benchmark/BenchmarkMetric+Defaults.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvQmVuY2htYXJrTWV0cmljK0RlZmF1bHRzLnN3aWZ0) | `36.79% <ø> (ø)` | | | [Sources/Benchmark/BenchmarkResult.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvQmVuY2htYXJrUmVzdWx0LnN3aWZ0) | `72.24% <100.00%> (-0.82%)` | :arrow_down: | | [Sources/Benchmark/Progress/Progress.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvUHJvZ3Jlc3MvUHJvZ3Jlc3Muc3dpZnQ=) | `57.41% <ø> (ø)` | | | [Sources/Benchmark/Progress/ProgressElements.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvUHJvZ3Jlc3MvUHJvZ3Jlc3NFbGVtZW50cy5zd2lmdA==) | `92.42% <ø> (ø)` | | | [Sources/Benchmark/Progress/Utilities.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvUHJvZ3Jlc3MvVXRpbGl0aWVzLnN3aWZ0) | `59.52% <ø> (ø)` | | | [Sources/Benchmark/Statistics.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvU3RhdGlzdGljcy5zd2lmdA==) | `72.03% <ø> (ø)` | | | [Tests/BenchmarkTests/BenchmarkResultTests.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-VGVzdHMvQmVuY2htYXJrVGVzdHMvQmVuY2htYXJrUmVzdWx0VGVzdHMuc3dpZnQ=) | `100.00% <100.00%> (ø)` | | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) | [Files](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) | Coverage Δ | | |---|---|---| | [Sources/Benchmark/BenchmarkClock.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvQmVuY2htYXJrQ2xvY2suc3dpZnQ=) | `33.72% <ø> (ø)` | | | [Sources/Benchmark/BenchmarkMetric+Defaults.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvQmVuY2htYXJrTWV0cmljK0RlZmF1bHRzLnN3aWZ0) | `36.79% <ø> (ø)` | | | [Sources/Benchmark/BenchmarkResult.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvQmVuY2htYXJrUmVzdWx0LnN3aWZ0) | `72.24% <100.00%> (-0.82%)` | :arrow_down: | | [Sources/Benchmark/Progress/Progress.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvUHJvZ3Jlc3MvUHJvZ3Jlc3Muc3dpZnQ=) | `57.41% <ø> (ø)` | | | [Sources/Benchmark/Progress/ProgressElements.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvUHJvZ3Jlc3MvUHJvZ3Jlc3NFbGVtZW50cy5zd2lmdA==) | `92.42% <ø> (ø)` | | | [Sources/Benchmark/Progress/Utilities.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvUHJvZ3Jlc3MvVXRpbGl0aWVzLnN3aWZ0) | `59.52% <ø> (ø)` | | | [Sources/Benchmark/Statistics.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-U291cmNlcy9CZW5jaG1hcmsvU3RhdGlzdGljcy5zd2lmdA==) | `72.03% <ø> (ø)` | | | [Tests/BenchmarkTests/BenchmarkResultTests.swift](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one#diff-VGVzdHMvQmVuY2htYXJrVGVzdHMvQmVuY2htYXJrUmVzdWx0VGVzdHMuc3dpZnQ=) | `100.00% <100.00%> (ø)` | | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one). Last update [02b4da9...1db45b5](https://app.codecov.io/gh/ordo-one/package-benchmark/pull/223?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ordo-one).

[github-actions[bot]]

Pull request had an unknown failure Pull request benchmark comparison [ubuntu-latest] with 'main' run at 2024-01-11T11:52:23+00:00