SwiftPM unconditionally rebuilds artefacts that should be cached for command plugins

[hassila]

Description

Some SwiftPM command plugin helper tools are desirable to build in release configuration as the processing they do is CPU-intensive - in my case, it's the BenchmarkTool of the Benchmark plugin which needs to do JSON parsing and the performance with optimisation turned on is just fine, but in debug mode it is not acceptable really.

Unfortunately, there seems to be a bug in SwiftPM that leads to a significant (full?) rebuild of everything if specifying the release configuration thusly:

swift package -c release benchmark list

For a normal run, we can see an initial long time (as the initial build of everything is done) and the command executes in 38 seconds:

hassila@ice ~/g/s/Benchmarks (main)> time swift package benchmark list
Building for debugging...
Build complete! (0.22s)
Building for debugging...
Build complete! (0.54s)
Build complete!
Building benchmark targets in release mode for benchmark run...
Building NIOPosixBenchmarks

Target 'NIOPosixBenchmarks' available benchmarks:
TCPEcho
TCPEchoAsyncChannel

________________________________________________________
Executed in   38.48 secs    fish           external
   usr time   76.25 secs  286.00 micros   76.25 secs
   sys time   21.90 secs  952.00 micros   21.90 secs

Then the second run runs fast in < 3 seconds.

hassila@ice ~/g/s/Benchmarks (main)> time swift package benchmark list
Building for debugging...
Build complete! (0.22s)
Building for debugging...
Build complete! (0.53s)
Build complete!
Building benchmark targets in release mode for benchmark run...
Building NIOPosixBenchmarks

Target 'NIOPosixBenchmarks' available benchmarks:
TCPEcho
TCPEchoAsyncChannel

________________________________________________________
Executed in    2.71 secs    fish           external
   usr time    1.96 secs  182.00 micros    1.96 secs
   sys time    0.54 secs  484.00 micros    0.54 secs

hassila@ice ~/g/s/Benchmarks (main)> history

Contrast this what happens if specifying the release configuration, first run is ~54 seconds:

hassila@ice ~/g/s/Benchmarks (main)> time swift package -c release benchmark list
Building for production...
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
[4/4] Compiling BenchmarkBoilerplateGenerator BenchmarkBoilerplateGenerator.swift
Build complete! (8.45s)
Building for production...
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
[12/12] Compiling Benchmark ARCStats.swift
Build complete! (24.70s)
Build complete!
Building benchmark targets in release mode for benchmark run...
Building NIOPosixBenchmarks

Target 'NIOPosixBenchmarks' available benchmarks:
TCPEcho
TCPEchoAsyncChannel

________________________________________________________
Executed in   53.84 secs    fish           external
   usr time   53.41 secs    0.18 millis   53.41 secs
   sys time    3.19 secs    1.03 millis    3.19 secs

But also the second run is ~54 seconds (!) - it performs a full rebuild here:

hassila@ice ~/g/s/Benchmarks (main)> time swift package -c release benchmark list
Building for production...
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
[4/4] Compiling BenchmarkBoilerplateGenerator BenchmarkBoilerplateGenerator.swift
Build complete! (8.44s)
Building for production...
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
[12/12] Compiling Benchmark ARCStats.swift
Build complete! (24.82s)
Build complete!
Building benchmark targets in release mode for benchmark run...
Building NIOPosixBenchmarks

Target 'NIOPosixBenchmarks' available benchmarks:
TCPEcho
TCPEchoAsyncChannel

________________________________________________________
Executed in   53.60 secs    fish           external
   usr time   53.22 secs    0.20 millis   53.22 secs
   sys time    3.15 secs    1.15 millis    3.15 secs

hassila@ice ~/g/s/Benchmarks (main)> time swift package benchmark list

Expected behavior

I would expect that the second run with release mode turned on would be without a full rebuild...

It would also be nice with some way to set the command plugin supporting tool to be built with optimization turned on as another way to solve this, but I can't get to work with swiftSettings for whatever reason.

Actual behavior

A full rebuild is done

Steps to reproduce

I'd suggest to reproduce this with SwiftNIO:

1. gh repo clone apple/swift-nio
2. cd swift-nio/Benchmarks/
3. time swift package -c release benchmark list

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.9.0

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.87.3 Apple Swift version 5.9.2 (swiftlang-5.9.2.2.56 clang-1500.1.0.2.5)
Target: arm64-apple-macosx14.0
Darwin ice.local 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:18 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6000 arm64

[hassila]

(on a general sidenote, I'd expect many users of plugins to want them to be run in release mode as users, if they do any non-trivial processing it's nice to get optimization turned on)

[MaxDesiatov]

(on a general sidenote, I'd expect many users of plugins to want them to be run in release mode as users, if they do any non-trivial processing it's nice to get optimization turned on)

As I've posted here https://forums.swift.org/t/embedded-swift-running-on-the-raspberry-pi-pico/69001/5, to date there was no way to build just the plugins and macros in release and the rest of the project in debug. The whole build graph had a single BuildParameters value for everything. In recent commits we started separating build parameters for build tools (macros and plugins) from build parameters for destination products. Right now those are still initialized from the same values, but at least this separation is possible and much easier to expose to users of SwiftPM.

This is not directly relevant to the nature of the bug though. IIUC when building everything in release, full rebuilds should be avoided regardless of the build parameters separation I described.

[hassila]

Thanks @MaxDesiatov, agree it's not related to this bug per se, but it gives a glimpse of a brighter future when we can run our plugins/macros in release while the rest is in debug (preferably by default as release config really, I'd think users would want to opt-out rather than opt-in for that specific use-case).

[MaxDesiatov]

I'm personally also of an opinion that should be the default, but we may start with an opt-in approach, with something like an --experimental-tools-release-configuration flag to test how it works, when it's ready for testing in the first place. When we're sure it works well, we can consider making that the default.

[neonichu]

The build configuration has no bearing how we build plugins themselves btw, they are always built in debug mode. The tools built for use by plugins can already build in a separate build operation (which is even visible in the CLI output due to the multiple 'Build complete!' messages) for quite some time (at least Swift 5.9, I believe).

I'm not sure I agree with building macros always in release, there's a massive cost to building a release build of swift-syntax. When developing macros we were actually discussing the opposite which was always building them in debug mode.

[hassila]

For me the plugin can be debug, it's just the supporting tool that needs release. Appreciate the conundrum with swift syntax - but seems that might need addressing in some different way too as even in debug that extra build time is painful.

[hassila]

(anyway, if this issue is fixed, we could at least opt in to do everything in release mode - the benchmark command plugin also triggers build with the swiftpm api - in release mode - if that has any bearing)

[neonichu]

the benchmark command plugin also triggers build with the swiftpm api - in release mode - if that has any bearing

I think this might be a good hint, it could be that the build triggered this way is subtly different from the "real" one, leading to changing commandlines between the builds.

[hassila]

This is where we build the subset of targets we need in release mode (the benchmark executable targets only):

https://github.com/ordo-one/package-benchmark/blob/2d3544e9bab6b0d6d8ac7bf5087b39048ad7878a/Plugins/BenchmarkCommandPlugin/BenchmarkCommandPlugin.swift#L290

[hassila]

Could it be that

                let buildResult = try packageManager.build(
                    .product(target.name), 
                    parameters: .init(configuration: .release)
                )

Unconditionally perform an actual build and/or compiler invocations under some circumstances?

I'm trying to work around this by manually building the dependent tool in addition to the benchmark targets (which seems to work ok, modulo the unnecessary double build of it in debug mode that is never used), but I see now that swift-frontend and swiftc are invoked and that a small subset of file in the build directory was updated, specifically the following (for my benchmark targets):

    8 -rw-r--r--   1 hassila  staff     1456 Dec 21 10:28 P90AbsoluteThresholdsBenchmark.swiftsourceinfo
    8 -rw-r--r--   1 hassila  staff      147 Dec 21 10:28 P90AbsoluteThresholdsBenchmark.abi.json
    8 -rw-r--r--   1 hassila  staff     1376 Dec 21 10:28 HistogramBenchmark.swiftsourceinfo
    8 -rw-r--r--   1 hassila  staff      147 Dec 21 10:28 HistogramBenchmark.abi.json
    8 -rw-r--r--   1 hassila  staff     1368 Dec 21 10:28 BenchmarkDateTime.swiftsourceinfo
    8 -rw-r--r--   1 hassila  staff      147 Dec 21 10:28 BenchmarkDateTime.abi.json
  840 -rw-r--r--   1 hassila  staff   365702 Dec 21 10:28 description.json
    8 -rw-r--r--   1 hassila  staff     1932 Dec 21 10:28 Basic.swiftsourceinfo
    8 -rw-r--r--   1 hassila  staff      147 Dec 21 10:28 Basic.abi.json

These are unconditionally updated for me in that case, but also e.g.:

> ls -lsrt .build/release/BenchmarkTool.product/
total 48
48 -rw-r--r--  1 hassila  staff  21934 Dec 21 10:31 Objects.LinkFileList
> ls -lsrt .build/release/P90AbsoluteThresholdsBenchmark.product/
total 40
40 -rw-r--r--  1 hassila  staff  18909 Dec 21 10:31 Objects.LinkFileList
...

The actual binary products are untouched though, so it seems we build but don't link?:

10824 -rwxr-xr-x   1 hassila  staff  5541296 Dec 21 09:23 BenchmarkTool*
    0 drwxr-xr-x   3 hassila  staff       96 Dec 21 09:23 BenchmarkTool.dSYM/
    0 drwxr-xr-x   3 hassila  staff       96 Dec 21 09:23 plugins/
   48 -rw-r--r--   1 hassila  staff    22376 Dec 21 09:23 P90AbsoluteThresholdsBenchmark.swiftmodule
    8 -rw-r--r--   1 hassila  staff      420 Dec 21 09:23 P90AbsoluteThresholdsBenchmark.swiftdoc
    0 drwxr-xr-x   6 hassila  staff      192 Dec 21 09:23 P90AbsoluteThresholdsBenchmark.build/
 9096 -rwxr-xr-x   1 hassila  staff  4655584 Dec 21 09:23 P90AbsoluteThresholdsBenchmark*

(for reference, an original build was done at 09:23, the timestamp of 10:28 just shows that those files were regenerated even though no source files changed)

[hassila]

(perhaps these are two different issues, hard for me to tell)

[hassila]

Are there anything else I could do to help troubleshoot this, just let me know - it quite significantly impacts run times for larger benchmark suites so we'd be happy to help try to fix this if possible.

[furby-tm]

I would just like to add, I too have experienced full package rebuilds related to SwiftPM package plugins, in my experience; using a plugin with @stackotter's SwiftBundler with an invocation such as this one, which I (no longer, after narrowing it down to this issue) use:

swift package --disable-sandbox plugin bundler run -p macOS Kraken

Subsequent builds using that command has caused me to lose countless hours of development time, waiting approximately 25+ minutes between adding a line of code to my project while SwiftPM tragically kept rebuilding all dependencies (in my case; all of Pixar's USD and the many ASWF libraries in which USD depends on 😅).

The workaround I am using for now is to instead not use any SwiftPM plugins, as I have installed the bundler locally, the usage of package plugins appears to be quite problematic at the moment.