ordo-one / package-benchmark

Swift benchmark runner with many performance metrics and great CI support
Apache License 2.0
326 stars 25 forks source link

Report Swift toolchain version being used. #213

Open JaapWijnen opened 11 months ago

JaapWijnen commented 11 months ago

In using the benchmark package. We're sometimes running into improvements and/or regressions in performance due to an update in toolchain. Is there a way we could include the toolchain used in the benchmark results? That way it's easier to clarify certain results or exclude this as a variable when there's a performance change.

hassila commented 11 months ago

One approach is to do what eg. SwiftNIO does and separate threshold results per toolchain as it’s built - it’s probably the most robust approach in general, WDYT?

hassila commented 11 months ago

See https://github.com/apple/swift-nio/tree/main/Benchmarks/Thresholds

hassila commented 3 months ago

See also https://github.com/swiftlang/swift/issues/75998