Missing required module '_FoundationCollections' when building with the Static Linux SDK

[barnard-b]

When building a Swift package using the September 11, 2024 Swift 6 toolchain and matching Static Linux SDK, compilation fails with error: missing required module '_FoundationCollections'

% xcrun --toolchain swift swift build --swift-sdk aarch64-swift-linux-musl   
Building for debugging...
error: emit-module command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'

System info

OS: macOS 14.6.1 Xcode: 16.0 public release, 16.0 release candidate, 15.4 Swift: Apple Swift version 6.0-dev (LLVM fc1036139c5e562, Swift 5bbf725fd958d34) Target: arm64-apple-macosx14.0 Static Linux SDK swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-11-a_static-linux-0.0.1

Package

Package.swift

// swift-tools-version: 6.0

import PackageDescription

let package = Package(
    name: "FoundationTest",
    targets: [
        .executableTarget(name: "FoundationTest")
    ]
)

Sources/FoundationTest/FoundationTest.swift

import Foundation

@main
struct App {
    static func main() async throws {
        print("Hello World!")
    }
}

[jmschonfeld]

@etcwilde @al45tair any chance you have any insight on what's going on here? For reference, in the standard Linux toolchain we have:

/usr/lib/swift_static/linux/_FoundationCollections.swiftmodule
/usr/lib/swift_static/linux/lib_FoundationCollections.a
/usr/lib/swift/linux/_FoundationCollections.swiftmodule

(in the non-static build, lib_FoundationCollections is statically linked into libFoundationEssentials.so)

but in the static swift Linux SDK, I don't see lib_FoundationCollections.a or _FoundationCollections.swiftmodule, but I do see

musl-1.2.5.sdk/x86_64/usr/lib/swift_static/linux-static/FoundationEssentials.swiftmodule
musl-1.2.5.sdk/x86_64/usr/lib/swift_static/linux-static/libFoundationEssentials.a

Do we have an issue of the packaging of the static SDK where we're missing these files, or is there an issue on the CMake side with swift-collections (that CMake file is at https://github.com/apple/swift-collections/blob/main/Sources/CMakeLists.txt and COLLECTIONS_SINGLE_MODULE and COLLECTIONS_FOUNDATION_TOOLCHAIN_MODULE are defined by swift-foundation)

[finagolfin]

I can reproduce with the just-released static linux SDK 6.0 when simply trying to build swift-argument-parser on Fedora 40 x86_64:

# This file requires XCTest, which isn't in the static linux SDK
> rm Sources/ArgumentParserTestHelpers/TestHelpers.swift
> ../swift-6.0-RELEASE-fedora39/usr/bin/swift build --swift-sdk x86_64-swift-linux-musl
Building for debugging...
error: emit-module command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'
<unknown>:0: error: missing required module '_FoundationCollections'

Simply building with the regular glibc-based static stdlib works fine though, ../swift-6.0-RELEASE-fedora39/usr/bin/swift build --static-swift-stdlib, as it has those missing _FoundationCollections files.

Looking at Evan's last change to the static linux SDK build script to add support for this Foundation rewrite, he added two CMake flags for this, but Jeremy previously added a couple more to the official build-script.

@jmschonfeld, is the _SwiftCollections_SourceDIR flag required for these files to be installed?

[al45tair]

I think @etcwilde is looking at this, but my guess is that the static SDK build script needs tweaking slightly to get CMake to install the additional library files.

[jmschonfeld]

@jmschonfeld, is the _SwiftCollections_SourceDIR flag required for these files to be installed?

Foundation needs some copy of swift-collections to build, but the parameters determine where that source copy comes from. With _SwiftCollections_SourceDIR provided (like in build-script-impl/build.ps1) it will build the swift-collections copy at the given path. Without that parameter provided, it will checkout the necessary tag from GitHub, clone a local copy, and build that copy. In a full toolchain/SDK build environment, I'd expect it to pass _SwiftCollections_SourceDIR but it indeed looks like that's missing from the linked build.sh. However I wouldn't expect that to cause this failure, since it'll just clone a copy from GitHub. Sounds like that's a separate, unrelated issue.

[finagolfin]

Foundation needs some copy of swift-collections to build

Right, but that wasn't the question, ie is it "required for these files to be installed" in the final static SDK? I figured it's somehow getting the source and building it, otherwise the dependency wouldn't be fulfilled and you'd get a compilation failure, but perhaps the variable has to be set for CMake to then finally install that static library and modules?

[jmschonfeld]

perhaps the variable has to be set for CMake to then finally install that static library and modules

No, the variable strictly defines the location of the source checkout for swift-collections

[al45tair]

Having said that I thought @etcwilde was working on it, I went looking. The problem is that swift-collections is trying to work out the system name by lowercasing the CMAKE_SYSTEM_NAME, but that fails for the Static SDK for Linux because the system name there needs to be linux-static. The other Foundation components accept that via the CMake variable SWIFT_SYSTEM_NAME, which gets set by Swift when it's doing the build, but swift-collections ignores that.

I'll have some PRs up tomorrow my time to fix that and a couple of other things I found along the way.

[jmschonfeld]

@al45tair should we move this issue over to the swift-collections repo if we need to make the CMake fix there?

cc @lorentey since this likely entails tagging a new swift-collections version for the toolchain build to pickup with CMake-only changes

[al45tair]

Yes, this probably should be on swift-collections. The other issues I found weren't related to this, but made getting to a fix take a little longer than necessary; I'll deal with those separately.

[finagolfin]

@al45tair, others are reporting more errors like <unknown>:0: warning: libc not found for 'x86_64-swift-linux-musl'; C stdlib may be unavailable, apple/swift-nio#2886, and error: unableToFind(tool: "swift-autolink-extract"), maybe because they're wrongly using the Xcode toolchain?

You may also want to update the doc to use the 6.0 release SDK and add a doc step so they can check and make sure they're not using the Xcode toolchain inadvertently.

[al45tair]

The warning is a known issue with the modulewrap program (rdar://115918181); it's harmless, but annoying. (Update: I just raised a PR to get rid of it)

The missing swift-autolink-extract is almost certainly, as you say, because they're using the Xcode toolchain. Updating the documentation isn't a bad idea, but let's make sure everything works first.

[doozMen]

The thing I had was that I overlooked the line on https://www.swift.org/documentation/articles/static-linux-getting-started.html

You cannot use the toolchain provided with Xcode to build programs using the SDK.

So you have to follow the instructions to build using the toolchain downloaded and not build xcode. That is indeed what solved the swift-autolink-extract.

But importing Foundation is an issue. It does not work as documented like stated in this post https://forums.swift.org/t/using-swift-foundation-with-linux-static-sdk/74642

Also doing import FoundationEssentials with the 6.0 toolchain from cli or from xcode when that toolchain is selected gives error No such module 'FoundationEssentials', on macOS 15.0 and on linux.

So also the docs on apple/swift-foundation that state it should be included are wrong. Any help why foundation is missing?

[al45tair]

So also the docs on apple/swift-foundation that state it should be included are wrong. Any help why foundation is missing?

Foundation itself is not missing, but _FoundationCollections is, which breaks things (_FoundationCollections is a dependency of FoundationEssentials, FWIW). This is fixed by https://github.com/apple/swift-collections/pull/421, but that will need to work its way through to a release in order to cure this problem.

[finagolfin]

Also doing import FoundationEssentials with the 6.0 toolchain from cli or from xcode when that toolchain is selected gives error No such module 'FoundationEssentials', on macOS 15.0 and on linux.

Hmm, that is new. Can you try something like the following two commands on your linux toolchain and report the results?

> ./swift-6.0-RELEASE-fedora39/usr/bin/swift -version
> find ./swift-6.0-RELEASE-fedora39/usr/lib/swift/linux -name "*.swiftdoc"

[shahmishal]

Swift 6.0 Nightly should resolve this issue with temporary workaround in the build script. error: missing required module '_FoundationCollections'

We would like to hear feedback if this resolves the issue for you.

Toolchain: https://download.swift.org/swift-6.0-branch/xcode/swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a/swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a-osx.pkg Static SDK: https://download.swift.org/swift-6.0-branch/static-sdk/swift-6.0-DEVELOPMENT-SNAPSHOT-[…]PSHOT-2024-09-17-a_static-linux-0.0.1.artifactbundle.tar.gz (b34ae1758ba66f2cd0d8c7e93b504df199905ddd69dabc7e6b686542b2c43547)

Note: We should not close this issue until the proper fix is merged into swift-collections repo.

[doozMen]

Also doing import FoundationEssentials with the 6.0 toolchain from cli or from xcode when that toolchain is selected gives error No such module 'FoundationEssentials', on macOS 15.0 and on linux.

Hmm, that is new. Can you try something like the following two commands on your linux toolchain and report the results?

> ./swift-6.0-RELEASE-fedora39/usr/bin/swift -version
> find ./swift-6.0-RELEASE-fedora39/usr/lib/swift/linux -name "*.swiftdoc"

Not sure I understand the find, this does not fin anything but the swift --version gives this swift --version Apple Swift version 6.0 (swift-6.0-RELEASE) Target: arm64-apple-macosx15.0

[doozMen]

@shahmishal will try with the de

Swift 6.0 Nightly should resolve this issue with temporary workaround in the build script. error: missing required module '_FoundationCollections'

We would like to hear feedback if this resolves the issue for you.

Toolchain: https://download.swift.org/swift-6.0-branch/xcode/swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a/swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a-osx.pkg Static SDK: https://download.swift.org/swift-6.0-branch/static-sdk/swift-6.0-DEVELOPMENT-SNAPSHOT-[…]PSHOT-2024-09-17-a_static-linux-0.0.1.artifactbundle.tar.gz (b34ae1758ba66f2cd0d8c7e93b504df199905ddd69dabc7e6b686542b2c43547)

Note: We should not close this issue until the proper fix is merged into swift-collections repo.

will try using those now and report back ...

[doozMen]

I think I'm mixing 2 issues that should be separated.

1. the static linux build seams to not work when Foundation is imported
2. the normal build of any project that imports Foundation does not build when swift 6 release of swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a-osx.pkg is used

This is for a very simple executable project that I mention in the ticket in Foundation. That has been closed since it seams to be a darwin related issue

https://github.com/apple/swift-foundation/issues/920

the reproducable code to try is

export TOOLCHAINS=$(plutil -extract CFBundleIdentifier raw /Library/Developer/Toolchains/swift-6.0-RELEASE.xctoolchain/Info.plist)
mkdir swift-tool && cd swift-tool
swift package init --type executable
echo `import Foundation` >> Sources/main.swift
swift build

This both with with release and the swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a.xctoolchain

[shahmishal]

Correct there are two issues:

1. static sdk running into error: missing required module '_FoundationCollections'
   • Fixed in swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a nightly tag.
   • To verify this fix, you will need to use Xcode 16 beta 4 or Xcode 15.x otherwise you will run into issue number 2.
2. Unable to build with Xcode 16 + Swift.org toolchain due to Darwin module
   • Fixed with https://github.com/swiftlang/swift/pull/76552, however nightly toolchain has not picked up this fix yet.

[finagolfin]

Not sure I understand the find, this does not fin anything

Nobody else has reported the No such module 'FoundationEssentials' error that you saw, so I'm trying to figure out if that's a third problem, particularly on linux. Could you detail how you see that when building for linux, ie what code you're compiling and what toolchain and SDK you're using?

I have seen something similar when cross-compiling the 6.0 corelibs for Android and trying to use them, so I'm trying to figure out if you're hitting that on certain linux configurations also.

[barnard-b]

Swift 6.0 Nightly should resolve this issue with temporary workaround in the build script. error: missing required module '_FoundationCollections'

We would like to hear feedback if this resolves the issue for you.

Toolchain: https://download.swift.org/swift-6.0-branch/xcode/swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a/swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a-osx.pkg Static SDK: https://download.swift.org/swift-6.0-branch/static-sdk/swift-6.0-DEVELOPMENT-SNAPSHOT-[…]PSHOT-2024-09-17-a_static-linux-0.0.1.artifactbundle.tar.gz (b34ae1758ba66f2cd0d8c7e93b504df199905ddd69dabc7e6b686542b2c43547)

@shahmishal When I attempt to install the new Static SDK it says the checksum does not match. I get the same different hash when downloading the file from https://www.swift.org/install/macos/ in Safari and running shasum

% shasum -a 256 ./swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a_static-linux-0.0.1.artifactbundle.tar.gz
83a88650cd0675552ce2cf8159c31966fde73418f49493c1644073fffe8be9f4

% xcrun --toolchain swift swift sdk install https://download.swift.org/swift-6.0-branch/static-sdk/swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a/swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a_static-linux-0.0.1.artifactbundle.tar.gz --checksum "b34ae1758ba66f2cd0d8c7e93b504df199905ddd69dabc7e6b686542b2c43547"
Downloading a Swift SDK bundle archive from `https://download.swift.org/swift-6.0-branch/static-sdk/swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a/swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a_static-linux-0.0.1.artifactbundle.tar.gz`...
                                                                        Downloading
100% [=======================================================================================================================================]
Downloading swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a_static-linux-0.0.1.artifactbundle.tar.gz

Swift SDK bundle archive successfully downloaded from `https://download.swift.org/swift-6.0-branch/static-sdk/swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a/swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a_static-linux-0.0.1.artifactbundle.tar.gz`.
Verifying if checksum of the downloaded archive is valid...
Error: Computed archive checksum `83a88650cd0675552ce2cf8159c31966fde73418f49493c1644073fffe8be9f4` does not match the provided checksum `b34ae1758ba66f2cd0d8c7e93b504df199905ddd69dabc7e6b686542b2c43547`.

[shahmishal]

checksum is generated by swift package compute-checksum command, its not the same as SHA256.

% swift package compute-checksum swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a_static-linux-0.0.1.artifactbundle.tar.gz
83a88650cd0675552ce2cf8159c31966fde73418f49493c1644073fffe8be9f4

[barnard-b]

checksum is generated by swift package compute-checksum command, its not the same as SHA256.

% swift package compute-checksum swift-6.0-DEVELOPMENT-SNAPSHOT-2024-09-17-a_static-linux-0.0.1.artifactbundle.tar.gz
83a88650cd0675552ce2cf8159c31966fde73418f49493c1644073fffe8be9f4

That is good to know. In this case though the output of swift package compute-checksum does not match the checksum I am getting from https://swift.org/install/macos/ (https://github.com/swiftlang/swift-org-website/commit/d80d00ce7e14da7b26d79e63627d623ef0cc4a1e) so swift sdk install is rejecting it.

[shahmishal]

The website issue should be resolved now, thanks for letting me know.

[barnard-b]

Thank you. The temporary workaround in the 9/17 toolchain and static SDK appears to be working, the test code compiles successfully.