Build self-contained Swift Android cross-compilation SDK artifactbundle

The Android SDK that is currently being generated by CI creates a destination.json file that can be used for cross-compilation to Android with a command like:

swift build --destination destination.json

An example destination.json that targets Android aarch64 from a Linux x86_64 host might look like:

{
  "version": 1,
  "target": "aarch64-unknown-linux-android24",
  "toolchain-bin-dir": "/home/runner/work/swift-android-sdk/swift-android-sdk/sdk-config/swift-5.10-RELEASE-ubuntu22.04/usr/bin",
  "sdk": "/usr/local/lib/android/sdk/ndk/26.2.11394342/toolchains/llvm/prebuilt/linux-x86_64/sysroot",
  "extra-cc-flags": [ "-fPIC", "-I/home/runner/work/swift-android-sdk/swift-android-sdk/sdk-config/swift-release-android-aarch64-24-sdk/usr/include" ],
  "extra-swiftc-flags": [
    "-resource-dir", "/home/runner/work/swift-android-sdk/swift-android-sdk/sdk-config/swift-release-android-aarch64-24-sdk/usr/lib/swift",
    "-tools-directory", "/usr/local/lib/android/sdk/ndk/26.2.11394342/toolchains/llvm/prebuilt/linux-x86_64/bin",
    "-Xcc", "-I/home/runner/work/swift-android-sdk/swift-android-sdk/sdk-config/swift-release-android-aarch64-24-sdk/usr/include",
    "-L/home/runner/work/swift-android-sdk/swift-android-sdk/sdk-config/swift-release-android-aarch64-24-sdk/usr/lib"
  ],
  "extra-cpp-flags": [ "-lstdc++" ]
}

One shortcoming of the destination.json format is that it relies on hardcoded absolute paths for the various tools, which means that it would need to be created separately for each host installation.

The SE-0387 – Swift SDKs for Cross-Compilation proposal, which has been accepted and implemented in Swift 5.10 and higher, allows for packaging all the SDK components into a single .artifactbundle which can be installed and managed using the swift sdk (or swift experimental-sdk for 5.10) command.

An SDK can be installed by manually unpacking an artifactbundle zip file, or can be installed like:

swift experimental-sdk install https://github.com/swift-android-sdk/swift-android-sdk/releases/download/5.10.1/swift-5.10.1-RELEASE_android-24-sdk.artifactbundle

Once installed, the SDKs can be listed:

zap ~ % swift experimental-sdk list
5.10-RELEASE_ubuntu_jammy_aarch64
5.10.1-RELEASE_android_24_aarch64
5.10.1-RELEASE_android_24_armv7
5.10.1-RELEASE_android_24_x86_64

After that, cross-compiling a Swift package is a simple matter of running:

swift build --experimental-swift-sdk 5.10.1-RELEASE_android_24_x86_64

Creating an artifactbundle would enable us to bundle the host toolchain, the Android NDK (or a subset thereof), and the Swift Android SDK for each architecture all together in a single artifact.

A 5.10.1-RELEASE_android_24.artifactbundle/info.json on macOS might look something like this:

{
  "schemaVersion": "1.0",
  "artifacts": {
    "5.10.1-RELEASE_android_24_x86_64": {
      "version": "0.0.1",
      "type": "swiftSDK",
      "variants": [
        {
          "supportedTriples": ["x86_64-apple-macosx", "arm64-apple-macosx"],
          "path": "5.10.1-RELEASE_android_24/x86_64-unknown-linux-android24"
        }
      ]
    },
    "5.10.1-RELEASE_android_24_aarch64": {
      "version": "0.0.1",
      "type": "swiftSDK",
      "variants": [
        {
          "supportedTriples": ["x86_64-apple-macosx", "arm64-apple-macosx"],
          "path": "5.10.1-RELEASE_android_24/aarch64-unknown-linux-android24"
        }
      ]
    },
    "5.10.1-RELEASE_android_24_armv7": {
      "version": "0.0.1",
      "type": "swiftSDK",
      "variants": [
        {
          "supportedTriples": ["x86_64-apple-macosx", "arm64-apple-macosx"],
          "path": "5.10.1-RELEASE_android_24/armv7-unknown-linux-android24"
        }
      ]
    }
  }
}

And a 5.10.1-RELEASE_android_24.artifactbundle/5.10.1-RELEASE_android_24/x86_64-unknown-linux-android24/toolset.json file might look like:

{
  "schemaVersion" : "1.0",
  "rootPath": "swift.xctoolchain/usr/bin",
  "swiftCompiler" : {
    "path": "swift.xctoolchain/usr/bin/swiftc",
    "extraCLIOptions" : [
      "-resource-dir", "../../../../swift-android.sdk/usr/lib/swift",
      "-L../../../../swift-android.sdk/usr/lib/x86_64-linux-android"
    ]
  },
  "cCompiler" : {
    "path": "../../../../android-ndk-darwin-x86_64/bin/clang",
    "extraCLIOptions" : [
      "-fPIC"
    ]
  },
  "cxxCompiler" : {
    "path": "../../../../android-ndk-darwin-x86_64/bin/clang++",
    "extraCLIOptions" : [
      "-lstdc++"
    ]
  },
  "linker" : {
    "path" : "../../../../android-ndk-darwin-x86_64/bin/ld.lld"
  },
  "librarian" : {
    "path" : "../../../../android-ndk-darwin-x86_64/bin/llvm-ar"
  },
  "debugger" : {
    "path" : "../../../../android-ndk-darwin-x86_64/bin/lldb"
  },
  "testRunner": {
    "path" : "../../../../scripts/run-swift-tests-on-emulator.sh"
  }
}

I've gotten an approximation of this setup working locally, and it is able to build various packages (swift-crypto, swift-collections, etc.). I was thinking that we could add a job to the CI that takes the outputs from the individual per-architecture toolchain builds and assembles them into artifactbundles that could then be surfaced as releases.

One example of such an assembly script can be found at static-linux/scripts/build.sh. Eventually, we might contribute it as a "recipe" in the swift-sdk-generator repo (e.g., LinuxRecipe.swift and WebAssemblyRecipe.swift), but a simple shell script is likely the easiest way to start.

finagolfin / swift-android-sdk

Build self-contained Swift Android cross-compilation SDK artifactbundle #163