Archive plugin execution error with AWS SDK

[fitomad]

Expected behavior

Create a folder with a zip file containing the project ready to deploy on AWS

Actual behavior

An error during the swift package archive command execution related to an AWS-SDK header file.

Command

swift package --disable-sandbox archive --output-path /Users/adolfo/Desktop --verbose 2

Output

-------------------------------------------------------------------------
configuration
-------------------------------------------------------------------------
{
  outputDirectory: /Users/adolfo/Desktop
  products: ["Favorites"]
  buildConfiguration: release
  baseDockerImage: swift:amazonlinux2
}
-------------------------------------------------------------------------
building "awslambdabackend" in docker
-------------------------------------------------------------------------
updating "swift:amazonlinux2" docker image
  amazonlinux2: Pulling from library/swift
  Digest: sha256:2d18edce3138194b7783aa75afb2d43712c536697cb7326a9fa2af43c91873df
  Status: Image is up to date for swift:amazonlinux2
  docker.io/library/swift:amazonlinux2
/usr/local/bin/docker run --rm -v /Users/adolfo/Documents/Globant/AWS Lambda Tech Talk/Lambdas/AWSLambdaBackend:/workspace -w /workspace swift:amazonlinux2 bash -cl swift build -c release --show-bin-path
  /workspace/.build/aarch64-unknown-linux-gnu/release
building "Favorites"
/usr/local/bin/docker run --rm -v /Users/adolfo/Documents/Globant/AWS Lambda Tech Talk/Lambdas/AWSLambdaBackend:/workspace -w /workspace swift:amazonlinux2 bash -cl swift build -c release --product Favorites --static-swift-stdlib
  error: 'aws-crt-swift': target 'S2N' has invalid header layout: umbrella header found at '/workspace/.build/checkouts/aws-crt-swift/aws-common-runtime/s2n/api/S2N.h', but directories exist next to it: /workspace/.build/checkouts/aws-crt-swift/aws-common-runtime/s2n/api/unstable; consider removing them
  [0/1] Planning build
error: ExitCode(rawValue: 1)
error: /usr/local/bin/docker run --rm -v /Users/adolfo/Documents/Globant/AWS Lambda Tech Talk/Lambdas/AWSLambdaBackend:/workspace -w /workspace swift:amazonlinux2 bash -cl swift build -c release --product Favorites --static-swift-stdlib failed with code 1

This the Package.swift file used to define the lambdas

import PackageDescription

let package = Package(
    name: "AWSLambdaTracking",
    platforms: [
        .macOS(.v12)
    ],
    dependencies: [
        .package(url: "https://github.com/swift-server/swift-aws-lambda-runtime.git", from: "1.0.0-alpha"),
        .package(url: "https://github.com/swift-server/swift-aws-lambda-events.git", branch: "main"),
        .package(url: "https://github.com/awslabs/aws-sdk-swift", exact: "0.10.0")
    ],
    targets: [
        // Targets are the basic building blocks of a package. A target can define a module or a test suite.
        // Targets can depend on other targets in this package, and on products in packages this package depends on.
        .executableTarget(
            name: "AWSLambdaTracking",
            dependencies: [
                .product(name: "AWSLambdaRuntime", package: "swift-aws-lambda-runtime"),
                .product(name: "AWSLambdaEvents", package: "swift-aws-lambda-events"),
                .product(name: "AWSDynamoDB", package: "aws-sdk-swift")
            ]),
        .testTarget(
            name: "AWSLambdaTrackingTests",
            dependencies: ["AWSLambdaTracking"]),
    ]
)

Steps to reproduce

1. Create a Swift package
2. Add the dependencies showed in the code snippet above
3. Create a Swift Lambda that inherits from LambdaHandler
4. Run swift build command. Results OK 👍
5. Run swift package --disable-sandbox archive --output-path /Users/adolfo/Desktop --verbose 2

If possible, minimal yet complete reproducer code (or URL to code)

Provided at the Actual Behavior section

SwiftAWSLambdaRuntime version/commit hash

Working with the 1.0.0-alpha version

Swift & OS version (output of swift --version && uname -a)

macOS version

Ventura 13.2.1

Swift version

swift-driver version: 1.62.15 Apple Swift version 5.7.2 (swiftlang-5.7.2.135.5 clang-1400.0.29.51)
Target: arm64-apple-macosx13.0
Darwin MacBook-Air-de-Adolfo.local 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:39:35 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T8103 arm64

[tomerd]

this looks to be an issue with aws-sdk-swift?

[tomerd]

cc @dave-moser

[fitomad]

That was my first impression, but building the solution works as expected (no errors). The error only appears when the archive operation was performed.

I also tested with different aws-sdk versions and the results after archiving were the same error.

But if I remove the aws-sdk from the package the archive operations ends as expected.

I think the error is a mix between the archive and the aws-sdk, even it's not an error, it's a kind of perfect storm building procedure.

[tomerd]

perhaps the docker image does not have all the required bits?

[fitomad]

Maybe, I didn't test with another docker image... 😮

[fitomad]

I have use the Amazon Linux 2 image for Swift 5.6 and 5.7.2, but the issue remains.

Inside my package I changed the swift tool version when launch tests agains Swift 5.6

// swift-tools-version: 5.6

Swift 5.6

Below this lines is the archive command setting Swift version to 5.6. The Amazon Linux 2 docker image was pulled and used as expected thanks to the new --swift-version argument 👍

swift package --disable-sandbox archive --output-path /Users/adolfo/Desktop --verbose 2 --swift-version 5.6

The command output is the following:

-------------------------------------------------------------------------
configuration
-------------------------------------------------------------------------
{
  outputDirectory: /Users/adolfo/Desktop
  products: ["Favorites"]
  buildConfiguration: release
  baseDockerImage: swift:5.6-amazonlinux2
}
-------------------------------------------------------------------------
building "awslambdabackend" in docker
-------------------------------------------------------------------------
updating "swift:5.6-amazonlinux2" docker image
  5.6-amazonlinux2: Pulling from library/swift
  Digest: sha256:450a6cbb1028de5947a49dc18bc9f4eece87c16432e12ca6073666401d9106b9
  Status: Image is up to date for swift:5.6-amazonlinux2
  docker.io/library/swift:5.6-amazonlinux2
/usr/local/bin/docker run --rm -v /Users/adolfo/Documents/Globant/AWS Lambda Tech Talk/Lambdas/AWSLambdaBackend:/workspace -w /workspace swift:5.6-amazonlinux2 bash -cl swift build -c release --show-bin-path
  /workspace/.build/aarch64-unknown-linux-gnu/release
building "Favorites"
/usr/local/bin/docker run --rm -v /Users/adolfo/Documents/Globant/AWS Lambda Tech Talk/Lambdas/AWSLambdaBackend:/workspace -w /workspace swift:5.6-amazonlinux2 bash -cl swift build -c release --product Favorites --static-swift-stdlib
  'swift-aws-lambda-events': error: package 'swift-aws-lambda-events' is using Swift tools version 5.7.0 but the installed version is 5.6.3
  'swift-aws-lambda-runtime'
  : 
  error: 
  package 'swift-aws-lambda-runtime' is using Swift tools version 5.7.0 but the installed version is 5.6.3
  'swift-aws-lambda-runtime': error: package 'swift-aws-lambda-runtime' is using Swift tools version 5.7.0 but the installed version is 5.6.3
  'swift-aws-lambda-events': error: package 'swift-aws-lambda-events' is using Swift tools version 5.7.0 but the installed version is 5.6.3
  'swift-aws-lambda-runtime': error: package 'swift-aws-lambda-runtime' is using Swift tools version 5.7.0 but the installed version is 5.6.3
  'swift-aws-lambda-events': error: package 'swift-aws-lambda-events' is using Swift tools version 5.7.0 but the installed version is 5.6.3
  'aws-crt-swift': error: target 'S2N' has invalid header layout: umbrella header found at '/workspace/.build/checkouts/aws-crt-swift/aws-common-runtime/s2n/api/S2N.h', but directories exist next to it: /workspace/.build/checkouts/aws-crt-swift/aws-common-runtime/s2n/api/unstable; consider removing them
error: /usr/local/bin/docker run --rm -v /Users/adolfo/Documents/Globant/AWS Lambda Tech Talk/Lambdas/AWSLambdaBackend:/workspace -w /workspace swift:5.6-amazonlinux2 bash -cl swift build -c release --product Favorites --static-swift-stdlib failed with code 1

There are errors related to the Swift tools version but the Umbrella header issue still remains.

Swift 5.7.2

Command:

swift package --disable-sandbox archive --output-path /Users/adolfo/Desktop --verbose 2 --swift-version 5.7.2

Output:

-------------------------------------------------------------------------
configuration
-------------------------------------------------------------------------
{
  outputDirectory: /Users/adolfo/Desktop
  products: ["Favorites"]
  buildConfiguration: release
  baseDockerImage: swift:5.7.2-amazonlinux2
}
-------------------------------------------------------------------------
building "awslambdabackend" in docker
-------------------------------------------------------------------------
updating "swift:5.7.2-amazonlinux2" docker image
  5.7.2-amazonlinux2: Pulling from library/swift
  Digest: sha256:4ac1263265156f50514e8d8e5162d042b8e7d914511b6be02fc1603afbac6745
  Status: Image is up to date for swift:5.7.2-amazonlinux2
  docker.io/library/swift:5.7.2-amazonlinux2
/usr/local/bin/docker run --rm -v /Users/adolfo/Documents/Globant/AWS Lambda Tech Talk/Lambdas/AWSLambdaBackend:/workspace -w /workspace swift:5.7.2-amazonlinux2 bash -cl swift build -c release --show-bin-path
  /workspace/.build/aarch64-unknown-linux-gnu/release
building "Favorites"
/usr/local/bin/docker run --rm -v /Users/adolfo/Documents/Globant/AWS Lambda Tech Talk/Lambdas/AWSLambdaBackend:/workspace -w /workspace swift:5.7.2-amazonlinux2 bash -cl swift build -c release --product Favorites --static-swift-stdlib
  error: 'aws-crt-swift': target 'S2N' has invalid header layout: umbrella header found at '/workspace/.build/checkouts/aws-crt-swift/aws-common-runtime/s2n/api/S2N.h', but directories exist next to it: /workspace/.build/checkouts/aws-crt-swift/aws-common-runtime/s2n/api/unstable; consider removing them
error: /usr/local/bin/docker run --rm -v /Users/adolfo/Documents/Globant/AWS Lambda Tech Talk/Lambdas/AWSLambdaBackend:/workspace -w /workspace swift:5.7.2-amazonlinux2 bash -cl swift build -c release --product Favorites --static-swift-stdlib failed with code 1

The Swift tool version error now is solved but the Umbrella header issue remains.

[tomerd]

I think we need someone from the aws-sdk project to weight in on this, seems like an issue compiling that library in the docker container

[fitomad]

Indeed.

Do you know if there is an Amazon engineer collaborating in this repo?

If no one is available maybe we can post this issue in the Swift Forums or open a new issue in the AWS-SDK repo

[dave-moser]

Agree, please open an issue in the aws-sdk-swift repo and an engineer can take a look: https://github.com/awslabs/aws-sdk-swift

[jagreenwood]

+1

Also seeing this issue:

/usr/local/bin/docker run --rm -v /Users/jeremy/Projects/xcode-cloud-lambda:/workspace -w /workspace swift:amazonlinux2 bash -cl swift build -c release --product XcodeCloudLambda --static-swift-stdlib failed with code 1

[fitomad]

The last news about this issue is that the issue opened at the aws-sdk-swift has been moved to the aws-crt-swift team.

The related issue is 👉 https://github.com/awslabs/aws-crt-swift/issues/163 and is planned for 2023 Q2.

[mattcorey]

Glad to see this is active on the aws-crt-swift project, and it's been successfully reproduced. Are there any known workarounds to this issue (are there any known older versions of the AWS libraries that work, for example), or should alternate libraries be evaluated in the meantime?

[fitomad]

Hi @mattcorey

I'm using Soto SDK and works really well.

[fitomad]

News from Amazon CRT team. The issue is solved at 0.12.0 release.

More info at 👉 https://github.com/awslabs/aws-crt-swift/issues/163