`SwiftTemplate` fails to build due to `Invalid manifest`

[markst]

We're using a XcodeBuildToolPlugin to generate mocks using Sourcery as mentioned in this issue: https://github.com/krzysztofzablocki/Sourcery/issues/1229

This works when running in Xcode, however occasionally when running tests using xcodebuild the following can occur:

⚠️ /Users/vagrant/Library/org.swift.swiftpm/configuration is not accessible or not writable, disabling user-level cache features.
⚠️ /Users/vagrant/Library/org.swift.swiftpm/security is not accessible or not writable, disabling user-level cache features.
⚠️ /Users/vagrant/Library/Caches/org.swift.swiftpm is not accessible or not writable, disabling user-level cache features.
❌ error: '2.2.2': Invalid manifest (compiled with: ["/Applications/Xcode-15.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc", "-vfsoverlay", "/var/folders/yy/6kcn9mkd5svdbqnznwf474f00000gn/T/TemporaryDirectory.rZXXGv/vfs.yaml", "-L", "/Applications/Xcode-15.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-lPackageDescription", "-Xlinker", "-rpath", "-Xlinker", "/Applications/Xcode-15.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-target", "arm64-apple-macosx13.0", "-sdk", "/Applications/Xcode-15.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk", "-F", "/Applications/Xcode-15.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks", "-I", "/Applications/Xcode-15.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-L", "/Applications/Xcode-15.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-swift-version", "5", "-I", "/Applications/Xcode-15.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-sdk", "/Applications/Xcode-15.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk", "-package-description-version", "5.7.0", "/Users/vagrant/Library/Developer/Xcode/DerivedData/Listen-fvqyfexjyixxxlfwlponyonxlklz/SourcePackages/plugins/Listen.output/ListenTests/SourceryPlugin/SourceryPlugin/Build/SwiftTemplate/8AA32FFF-75B5-408E-9C19-FCDB218A2B59/2.2.2/Package.swift", "-Xfrontend", "-disable-implicit-concurrency-module-import", "-Xfrontend", "-disable-implicit-string-processing-module-import", "-o", "/var/folders/yy/6kcn9mkd5svdbqnznwf474f00000gn/T/TemporaryDirectory.NscvFM/2.2.2-manifest"])
❌ error: fatalError

[markst]

This repeatedly occurs when attempting to build using Bitrise continuous integration platform

[markst]

We're currently using Sourcery 2.2.2.

I've attempting passing buildPath arguments to sourcery using both outputDirectory and NSTemporaryDirectory()

private func createBuildCommands(
    tool: PluginContext.Tool,
    configPath: Path,
    outputDirectory: Path
) -> [Command] {
    let cachePath = outputDirectory.appending(subpath: "SourceryPlugin/Cache")
    let buildPath = outputDirectory.appending(subpath: "SourceryPlugin/Build")
    return [
        .prebuildCommand(
            displayName: "Generate mocked types for target",
            executable: tool.path,
            arguments: [
                "--config", configPath,
                "--cacheBasePath", cachePath,
                "--buildPath", buildPath,
                "--disableCache",
                "--verbose"
            ],
            environment: ["DERIVED_SOURCES_DIR": outputDirectory],
            outputFilesDirectory: outputDirectory
        )

[markst]

I'm now struggling to reproduce locally, the output appears to succeed:

        Raw compilation of SwiftTemplate took: 7.120736837387085

[markst]

as a sidenote, the SwiftTemplate binary is built each time which leads me to think the executableCacheKey is changing despite no change in code?

[art-divin]

as a sidenote, the SwiftTemplate binary is built each time which leads me to think the executableCacheKey is changing despite no change in code?

Hi @markst , aside from other issues you've reported, for which there are billions of thanks ❤️ , SwiftTemplate binary is compiled once (in single threaded mode of running Sourcery), and then re-used. Other components are cached on per-execution basis.

You can verify this by using --verbose flag when running Sourcery starting 2.2.2 release. It is clearly shown when SwiftTemplate is reused from cache, and when it is re-compiled again due to a different cacheKey.

[swwol]

You can normally resolve this issue using swift package purge-cache

[markst]

@art-divin possibly it's failing to get the cached build since it's failing to complete the build?

[markst]

might be worth mentioning if I try to open the generated Package.swift in xcode it complains:

[art-divin]

Hello @markst ,

I wonder, can you try using Sourcery 2.2.1 version?

---

There is one concurrency issue to be resolved by the end of the upcoming week (next release should be done on the next weekend). Maybe it is related to the issue you are facing, but I suspect it is not. 🕵🏻‍♂️

[art-divin]

Investigating 🕵🏻

[art-divin]

We're currently using Sourcery 2.2.2.

I've attempting passing buildPath arguments to sourcery using both outputDirectory and NSTemporaryDirectory()

private func createBuildCommands(
    tool: PluginContext.Tool,
    configPath: Path,
    outputDirectory: Path
) -> [Command] {
    let cachePath = outputDirectory.appending(subpath: "SourceryPlugin/Cache")
    let buildPath = outputDirectory.appending(subpath: "SourceryPlugin/Build")
    return [
        .prebuildCommand(
            displayName: "Generate mocked types for target",
            executable: tool.path,
            arguments: [
                "--config", configPath,
                "--cacheBasePath", cachePath,
                "--buildPath", buildPath,
                "--disableCache",
                "--verbose"
            ],
            environment: ["DERIVED_SOURCES_DIR": outputDirectory],
            outputFilesDirectory: outputDirectory
        )

Hello @markst ,

first thing I notice, is that you are using --disableCache flag. When this flag is set, cacheKey is ignored and SwiftTemplate is recompiled for every run.

This is not the root cause of your issue, but thing is, --cacheBasePath is ignored and unused because of the --disableCache flag.

[art-divin]

@markst by the way, the speed with which in the gif you've shared the build is failing, tells me that the compilation of SwiftTemplate fails for one or the other reason. Flag verbose should expose some details about these processes, would you mind checking the raw log output in Xcode's Report Navigator for this exact build phase - plugin containing sourcery invocation?

That would help a lot 👍🏻