search

kkebo / zyphy

Zyphy is (or will be) a fast web browser engine written in Swift.
MIT License
4 stars 1 forks source link

ci: remove `set +e` from `swift package benchmark baseline compare` #121

Closed kkebo closed 1 month ago

kkebo commented 1 month ago

swift package benchmark baseline check may exit with status code 4 even if there are no performance regressions, but swift package benchmark baseline compare always returns status code 0 or 1.

github-actions[bot] commented 1 month ago

Code Metrics Report

main (48d5b3d) #121 (d548e16) +/-
Coverage 85.6% 85.5% -0.2%
Details ``` diff | | main (48d5b3d) | #121 (d548e16) | +/- | |-----------|----------------|----------------|-------| - | Coverage | 85.6% | 85.5% | -0.2% | | Files | 10 | 10 | 0 | | Lines | 1614 | 1607 | -7 | - | Covered | 1382 | 1374 | -8 | ```

Reported by octocov

github-actions[bot] commented 1 month ago

Benchmark Summary

Sat Oct 12 01:46:50 UTC 2024

New baseline 'pull_request' is WITHIN the 'main' baseline thresholds.

Comparing results between 'main' and 'pull_request'

Host 'd22bcd5f5866' with 4 'x86_64' processors with 15 GB memory, running:
#26~22.04.1-Ubuntu SMP Thu Jul 11 22:33:04 UTC 2024

MyBenchmark

lipsum metrics

Time (wall clock): results within specified thresholds, fold down for details.

| Time (wall clock) (μs) * | p0 | p25 | p50 | p75 | p90 | p99 | p100 | Samples | |:----------------------------------------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:| | main | 11 | 11 | 11 | 11 | 11 | 11 | 12 | 100 | | pull_request | 11 | 11 | 11 | 11 | 11 | 12 | 13 | 100 | | Δ | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | | Improvement % | 0 | 0 | 0 | 0 | 0 | -9 | -8 | 0 |

lipsum-zh metrics

Time (wall clock): results within specified thresholds, fold down for details.

| Time (wall clock) (ns) * | p0 | p25 | p50 | p75 | p90 | p99 | p100 | Samples | |:----------------------------------------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:| | main | 1771 | 1780 | 1783 | 1790 | 1795 | 1834 | 1862 | 100 | | pull_request | 1768 | 1776 | 1781 | 1787 | 1795 | 1833 | 1863 | 100 | | Δ | -3 | -4 | -2 | -3 | 0 | -1 | 1 | 0 | | Improvement % | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

medium-fragment metrics

Time (wall clock): results within specified thresholds, fold down for details.

| Time (wall clock) (μs) * | p0 | p25 | p50 | p75 | p90 | p99 | p100 | Samples | |:----------------------------------------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:| | main | 41 | 42 | 42 | 42 | 42 | 48 | 49 | 100 | | pull_request | 42 | 42 | 42 | 42 | 42 | 51 | 55 | 100 | | Δ | 1 | 0 | 0 | 0 | 0 | 3 | 6 | 0 | | Improvement % | -2 | 0 | 0 | 0 | 0 | -6 | -12 | 0 |

small-fragment metrics

Time (wall clock): results within specified thresholds, fold down for details.

| Time (wall clock) (ns) * | p0 | p25 | p50 | p75 | p90 | p99 | p100 | Samples | |:----------------------------------------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:| | main | 4633 | 4657 | 4665 | 4674 | 4682 | 4735 | 4785 | 100 | | pull_request | 4601 | 4628 | 4633 | 4645 | 4661 | 4768 | 5245 | 100 | | Δ | -32 | -29 | -32 | -29 | -21 | 33 | 460 | 0 | | Improvement % | 1 | 1 | 1 | 1 | 0 | -1 | -10 | 0 |

strong metrics

Time (wall clock): results within specified thresholds, fold down for details.

| Time (wall clock) (μs) * | p0 | p25 | p50 | p75 | p90 | p99 | p100 | Samples | |:----------------------------------------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:| | main | 20 | 20 | 20 | 20 | 20 | 21 | 21 | 100 | | pull_request | 20 | 20 | 20 | 20 | 21 | 21 | 23 | 100 | | Δ | 0 | 0 | 0 | 0 | 1 | 0 | 2 | 0 | | Improvement % | 0 | 0 | 0 | 0 | -5 | 0 | -10 | 0 |

tiny-fragment metrics

Time (wall clock): results within specified thresholds, fold down for details.

| Time (wall clock) (ns) * | p0 | p25 | p50 | p75 | p90 | p99 | p100 | Samples | |:----------------------------------------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:| | main | 446 | 449 | 450 | 451 | 459 | 480 | 502 | 100 | | pull_request | 454 | 455 | 456 | 457 | 464 | 482 | 512 | 100 | | Δ | 8 | 6 | 6 | 6 | 5 | 2 | 10 | 0 | | Improvement % | -2 | -1 | -1 | -1 | -1 | 0 | -2 | 0 |