SPM doesn't enumerate source files properly in different configurations

compnerd commented 1 year ago

Description

When building https://github.com/thebrowsercompany/swift-composable-architecture in release mode, the generated OFM for OpenCombine is invalid, e.g.

Building for production...
<unknown>:0: error: supplementary output file map 'C:\\Users\\abdulras\\AppData\\Local\\Temp\\TemporaryDirectory.T3w0D9\\supplementaryOutputs-1' is missing an entry for 'S:\\SourceCache\\thebrowsercompany\\swift-composable-architecture\\.build\\checkouts\\OpenCombine\\Sources\\OpenCombine\\AnyCancellable.swift' (this likely indicates a compiler issue; please submit a bug report (https://swift.org/contributing/#reporting-bugs))

Note that building OpenCombine directly does not exhibit this issue.

Expected behavior

The internal generated state would be correct.

Actual behavior

The internal generated state is incorrect.

Steps to reproduce

git clone https://github.com/thebrowsercompany/swift-composable-architecture
cd swift-composable-architecture
git checkout develop
swift build -c release

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.10.0-dev

Swift & OS version (output of `swift --version ; uname -a`)

Swift version 5.11-dev (LLVM 61776217335130e, Swift 4bcc25e99094074) Target: x86_64-unknown-windows-msvc

neonichu commented 1 year ago

I don't believe SwiftPM creates supplementary output file maps but the driver does.

@shahmishal could we move this issue to the driver repo?

mstokercricut commented 7 months ago

FYI, we're seeing this issue building a Swift codebase for Windows with Swift 5.9.1. We're seeing the same issue when building for Linux and Android. It is a library that depends on a stack of others: C -> B -> A -> Swift runtime. B & A alone build just fine, and other configurations of C do as well (for macOS/iOS & Wasm). All together, they have ~40k lines of Swift and a few hundred files.

Any movement on this? Anyhow I could help by testing against our codebase?

mstokercricut commented 7 months ago

Also, looking for more info on this, I just tried building swift-composable-architecture on Windows 11 with Swift 5.9.1 and encountered the same issue as @compnerd

Interestingly, running this with --verbose, the command that causes the error is the only one executed up to that point that passes an arguments file to swift-frontend.exe:

C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swift-frontend.exe @C:\Users\mstoker\AppData\Local\Temp\TemporaryDirectory.u5IFhQ\arguments-619307035247028663.resp

:0: error: supplementary output file map 'C:\\Users\\mstoker\\AppData\\Local\\Temp\\TemporaryDirectory.u5IFhQ\\supplementaryOutputs-1' is missing an entry for 'C:\\Users\\mstoker\\Desktop\\swift-composable-architecture\\.build\\checkouts\\OpenCombine\\Sources\\OpenCombine\\AnyCancellable.swift' (this likely indicates a compiler issue; please submit a bug report (https://swift.org/contributing/#reporting-bugs)) This is also the case with our codebase. The swift-frontend.exe command that fails is the first in the build that uses an arguments file.

mstokercricut commented 7 months ago

Concatenating the files in the codebase into one file allows the build to complete without issue. I suspect that there is a bug of some kind with handling of arguments passed via file to swift-frontend.exe on Windows.

@compnerd as a workaround, reducing the number of files (or the length of their names) may allow you to build swift-composable-architecture on Windows as well.

mstokercricut commented 5 months ago

@compnerd we've hit this bug in a few more contexts, each time it is when the command line being passed to swift-frontend.exe is long enough that it is being passed as a file. The problem likely lies there. Is there any debug logging I can provide to help with the fix?

compnerd commented 5 months ago

I don't remember enough of the details about this to know what debug information would be useful. I don't know if the OFM is missing entries or is getting truncated. The original error seems to indicate that there is something missing in the mapping itself.

tristanlabelle commented 1 month ago

I hit this with a module containing ~117 files (still below the default threshold) and attached a debugger to stop when swift-frontend is being invoked. This is the supplementary output file map on disk:

"D:\\swift-winrt\\build\\debug\\Generator\\InteropTests\\WinRTComponent\\Projection\\Sources\\WinRTComponent\\Projection\\WinRTComponent\\Arrays.swift":
  dependencies: "D:\\swift-winrt\\build\\debug\\Generator\\InteropTests\\WinRTComponent\\Projection\\Build\\WinRTComponent\\Projection\\WinRTComponent.d"
  swiftsourceinfo: "D:\\swift-winrt\\build\\debug\\Generator\\InteropTests\\WinRTComponent\\Projection\\Build\\WinRTComponent\\Projection\\WinRTComponent.swiftsourceinfo"
  swiftmodule: "D:\\swift-winrt\\build\\debug\\Generator\\InteropTests\\WinRTComponent\\Projection\\Build\\WinRTComponent\\Projection\\WinRTComponent.swiftmodule"
  swiftdoc: "D:\\swift-winrt\\build\\debug\\Generator\\InteropTests\\WinRTComponent\\Projection\\Build\\WinRTComponent\\Projection\\WinRTComponent.swiftdoc"

From code inspection, it looks like it was supposed to be json, but someone ate the curly braces.

And the compiler says:

<unknown>:0: error: supplementary output file map 'C:\\Users\\tristan\\AppData\\Local\\Temp\\TemporaryDirectory.GHm1Rw\\supplementaryOutputs-1' is missing an entry for 'D:\\swift-winrt\\build\\debug\\Generator\\InteropTests\\WinRTComponent\\Projection\\Sources\\WinRTComponent\\Projection\\WinRTComponent\\Arrays.swift' (this likely indicates a compiler issue; please submit a bug report (https://swift.org/contributing/#reporting-bugs))

Curiously that Array.swift is the only one present in the supplementary output file map.

tristanlabelle commented 1 month ago

Ah never mind, it's yaml. Then I don't understand why there is a problem, but here are the files from my repro:

arguments-2560280406336689506.resp.txt supplementaryOutputs-1.txt

swiftlang / swift-driver