Docker Builds specifying arm64 fail to build on x86 hardware

[gestrich]

Describe the bug

I've recently switched to building swift executables for Graviton2 on AWS. This requires building an arm64 docker image, rather than x86. This works fine on both my M1 and x86 (ex Building the Lambda runtime). However, I get failures when I depend on the Soto package (also get the issue with AWS-SDK-Swift from awslabs).

To Reproduce

I have a sample branch that demonstrates this: https://github.com/gestrich/Swift-Lambda-M1/tree/soto-sample

1. Clone the above repo on an x86 based mac.
2. Checkout the soto-sample branch.
3. Run make build-HomeAPI to trigger the docker build process.
4. Error will be hit:

   Working copy of https://github.com/swift-server/async-http-client.git resolved at 1.10.0
   Creating working copy for https://github.com/soto-project/soto.git
   qemu: uncaught target signal 5 (Trace/breakpoint trap) - core dumped
   /usr/bin/bash: line 1:   735 Trace/breakpoint trap   swift build --product HomeAPI -c release --build-path /build-target
   make: *** [builder-bot] Error 133

Expected behavior

Expect the image to create a Swift executable. By removing Soto as a package dependency, the build will be successful.

Also, you can work around the issue by editing the Makefile to build an x86 image, rather than arm64.

Setup (please complete the following information):

• OS: macOS Monterey 12.3.1
• Version of soto: 5.13.1
• Authentication mechanism: N/A

[gestrich]

Update: So bumping the Docker memory from 2gb to 25gb helped get the build along further.

I still cannot successful build the Swift executable though after depending on the soto package. It gets close to the end of build process and just hangs. I tried this a couple of times wit the same result. Here is where it stopped over night.

...
[872/877] Compiling a_strex.c
[873/877] Compiling a_bitstr.c
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization[874/878] Compiling NIOCore AddressedEnvelope.swift
remark: Incremental compilation has been disabled: it is not compatible with whole module optimizationremark: Incremental compilation has been disabled: it is not compatible with whole module optimization[876/881] Compiling NIOEmbedded Embedded.swift
[878/882] Compiling Crypto AES-GCM.swift
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization[880/884] Compiling SotoCrypto ByteArray.swift
[882/885] Compiling NIOPosix BSDSocketAPICommon.swift
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization[884/887] Compiling NIO Exports.swift
remark: Incremental compilation has been disabled: it is not compatible with whole module optimizationremark: Incremental compilation has been disabled: it is not compatible with whole module optimizationremark: Incremental compilation has been disabled: it is not compatible with whole module optimizationremark: Incremental compilation has been disabled: it is not compatible with whole module optimization[886/892] Compiling NIOFoundationCompat ByteBuffer-foundation.swift
[888/893] Compiling NIOTLS ApplicationProtocolNegotiationHandler.swift
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization[890/895] Compiling NIOSOCKS SOCKSClientHandler.swift
[892/896] Compiling NIOTransportServices NIOFilterEmptyWritesHandler.swift
[894/897] Compiling NIOHTTP1 ByteCollectionUtils.swift
remark: Incremental compilation has been disabled: it is not compatible with whole module optimizationremark: Incremental compilation has been disabled: it is not compatible with whole module optimizationremark: Incremental compilation has been disabled: it is not compatible with whole module optimization[896/901] Compiling NIOHTTPCompression HTTPCompression.swift
[898/902] Compiling SotoSignerV4 credentials.swift
[900/903] Compiling NIOHPACK DynamicHeaderTable.swift
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization[902/905] Compiling NIOHTTP2 ConnectionStateMachine.swift

[0xTim]

This will be a compiler error rather than a Soto error. Soto is big enough that when running the compiler in emulation mode it just baulks. You probably need to raise this on https://github.com/apple/swift.

To fix this, how are you building the executable? i.e. on what OS/architecture, what Docker image and what toolchain etc

[gestrich]

Thanks @0xTim

You can find more info in my sample app that demonstrates this. See the Dockerfile, Makefile & Package.swift but here are some high-level details:

• Docker Image: swift:5.6.1-amazonlinux2
• Swift Toolchain: 5.5
• OS: macOS Monterey 12.3.1

[0xTim]

So just to confirm @gestrich you're building on an M1 with 5.6 AL2 and the arm platform tag and it's still running in emulation mode?

[gestrich]

This issue is only encountered when building on an Intel machine and an arm64 image. When I build on my M1, I don't have any issue and arm64 images will not be built in emulation mode -- so it is much faster.

A specific use-case to understand the motivation for sharing this issue: If someone wants to use AWS Graviton2 (arm64), they will need to build on an M1 (if building locally). Intel machines will be blocked from what I can tell from building for Graviton2.

Let me know if that makes sense. We can close this issue and I can reopen with Swift.

[0xTim]

Ahhh I see, yeah best raising on the Swift repo. It will either need fixing there or in Docker.

I guess the best workaround for now is CodeBuild with a Graviton machine, though I think M1 runners for GH actions are currently in a private beta, so hopefully not too far out!