search

ordo-one / package-benchmark

Swift benchmark runner with many performance metrics and great CI support
Apache License 2.0
326 stars 25 forks source link

Skip compilation when there's no code changes #296

Open amomchilov opened 2 weeks ago

amomchilov commented 2 weeks ago

Hey there,

It looks like the plugin does a full rebuild on every command, even when no code has changed. E.g. on the sample project, running swift package benchmark baseline list takes ~30 seconds, even on a repeated invocation.

$ time swift package benchmark baseline list
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'BenchmarkBoilerplateGenerator' complete! (0.66s)
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'BenchmarkBoilerplateGenerator' complete! (0.12s)
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'BenchmarkBoilerplateGenerator' complete! (0.12s)
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'BenchmarkBoilerplateGenerator' complete! (0.11s)
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'BenchmarkBoilerplateGenerator' complete! (0.12s)
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'BenchmarkBoilerplateGenerator' complete! (0.12s)
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'BenchmarkBoilerplateGenerator' complete! (0.12s)
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'BenchmarkTool' complete! (1.83s)
Building benchmark targets in release mode for benchmark run...
Building Samples
Building Miscellaneous
Building Minimal
Building MemoryTwo
Building MemoryOne
Building Foundation-Benchmark
Building External-Benchmarks
Build complete!
swift package benchmark baseline list  45.83s user 4.07s system 160% cpu 31.024 total

I'd be happy to contribute to fixing this, if you could point me in the right directiont ot start.

MahdiBM commented 2 weeks ago

I wonder if a --skip-build flag can be of use, it can be directly passed to swift build by the benchmarking package. I think It'll be valuable ... we can skip some CI time when we know that we have build the package in the previous step and there are no changes.

hassila commented 2 weeks ago

Yeah, this is frustrating - I believe this is a possible workaround as pointed out by @MahdiBM :

https://github.com/swiftlang/swift-package-manager/issues/7210#issuecomment-2440097567

(that case is about this specific problem, there should be no rebuild really - if that SPM case is fixed, it should "just work" AFAICT)

Having a skip flag is problematic, as the discovery of the benchmarks are done using the SPM API (we get the path to the executables from there).