Can't package lambda for .NET 7 AOT from within a Docker Container

[Hammatt]

Describe the bug

Our CI build pipelines all run inside docker contains so that we can easily control versions of the toolkits that we are using.

It seems that the newest version of the Lambda build tookit tries to start up it's own docker container which doesn't seem to work very well when running already from within docker. This has worked great up until now, but the .NET 7 process seems to have broken.

Starting container for native AOT build using build image: public.ecr.aws/sam/build-dotnet7:latest-x86_64.
  Unknown error executing command: Failed to locate docker CLI executable. Make sure the docker CLI is installed in the environment PATH.
     at Amazon.Common.DotNetCli.Tools.DockerCLIWrapper..ctor(IToolLogger logger, String workingDirectory) in C:\build\src\Amazon.Common.DotNetCli.Tools\DockerCLIWrapper.cs:line 23
     at Amazon.Lambda.Tools.LambdaPackager.CreateApplicationBundle(LambdaToolsDefaults defaults, IToolLogger logger, String workingDirectory, String projectLocation, String configuration, String targetFramework, String msbuildParameters, String architecture, Boolean disableVersionCheck, LayerPackageInfo layerPackageInfo, Boolean isNativeAot, Nullable`1 useContainerForBuild, String containerImageForBuild, String codeMountDirectory, String& publishLocation, String& zipArchivePath) in C:\build\src\Amazon.Lambda.Tools\LambdaPackager.cs:line 144
     at Amazon.Lambda.Tools.Commands.PackageCommand.PerformActionAsync() in C:\build\src\Amazon.Lambda.Tools\Commands\PackageCommand.cs:line 231
     at Amazon.Common.DotNetCli.Tools.Commands.BaseCommand`1.ExecuteAsync() in C:\build\src\Amazon.Common.DotNetCli.Tools\Commands\BaseCommand.cs:line 47
  Process exited with code 255
  Process exited with code 255 (Step: Command Line)
  Step Command Line failed

Expected Behavior

The Amazon.Lambda.Tools toolkit can be ran from within a docker container

Current Behavior

the dotnet lambda package command crashes when ran in docker

Reproduction Steps

1. Create a .NET 7 lambda project using AOT following this AWS guide
2. docker run -v ${PWD}:/build -it mcr.microsoft.com/dotnet/sdk:7.0 sh
3. apt-get update && apt-get -y install zip
4. dotnet tool install -g Amazon.Lambda.Tools
5. export PATH="$PATH:/root/.dotnet/tools"
6. dotnet lambda package -pl build/Examples.AWS.Lambda.S3

Resulting Error

Amazon Lambda Tools for .NET Core applications (5.6.2)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Found /etc/os-release
Architecture not provided, defaulting to x86_64 for container build image.
Executing publish command
Starting container for native AOT build using build image: public.ecr.aws/sam/build-dotnet7:latest-x86_64.
Unknown error executing command: Failed to locate docker CLI executable. Make sure the docker CLI is installed in the environment PATH.
   at Amazon.Common.DotNetCli.Tools.DockerCLIWrapper..ctor(IToolLogger logger, String workingDirectory) in C:\build\src\Amazon.Common.DotNetCli.Tools\DockerCLIWrapper.cs:line 23
   at Amazon.Lambda.Tools.LambdaPackager.CreateApplicationBundle(LambdaToolsDefaults defaults, IToolLogger logger, String workingDirectory, String projectLocation, String configuration, String targetFramework, String msbuildParameters, String architecture, Boolean disableVersionCheck, LayerPackageInfo layerPackageInfo, Boolean isNativeAot, Nullable`1 useContainerForBuild, String containerImageForBuild, String codeMountDirectory, String& publishLocation, String& zipArchivePath) in C:\build\src\Amazon.Lambda.Tools\LambdaPackager.cs:line 144
   at Amazon.Lambda.Tools.Commands.PackageCommand.PerformActionAsync() in C:\build\src\Amazon.Lambda.Tools\Commands\PackageCommand.cs:line 231
   at Amazon.Common.DotNetCli.Tools.Commands.BaseCommand`1.ExecuteAsync() in C:\build\src\Amazon.Common.DotNetCli.Tools\Commands\BaseCommand.cs:line 47

Possible Solution

Consider not trying to spin up docker containers if the tool is already running in docker?

Additional Information/Context

No response

Targeted .NET platform

.NET 7

CLI extension version

amazon.lambda.tools 5.6.2

Environment details (OS name and version, etc.)

mcr.microsoft.com/dotnet/sdk:7.0 sh

[ashishdhingra]

Hi @Hammatt,

Good afternoon.

Thanks for reporting the issue. The referenced article/guide clearly mentions Docker as one of the prerequisites when using AWS .NET CLI Extensions tooling if compiling on a non-Amazon Linux 2 based machine. Correct me if I'm wrong, I guess you want to build docker image inside a Docker container, which is pretty unusual scenario when the tooling required Docker CLI to build the image. You might refer to https://forums.docker.com/t/how-can-i-run-docker-command-inside-a-docker-container/337/17 for your use case, but not sure if it would work.

Thanks, Ashish

[Hammatt]

Hi @ashishdhingra, thanks for the response.

Yes that's right, we are trying to run the dotnet lambda package command from within a docker container.

The concern around allowing docker commands from within docker containers is that this works around some of the isolation that our current CI system gives us.

I am not sure I would call running CI commands from within Docker an unusual scenario, it is extremely common (and recommended) within TeamCity as a way to control shared runtime environments.

I can have a look to see if we would be happy to enabling docker containers to run docker commands, but the preference would be for us to be able to run the tool from inside a docker container as we have done up until now.

The problem is that the interface that we were using has essentially been broken by requiring Docker. Does it really require docker? What are you using it for? Is it just to provide some isolation and make sure you have all of your other prerequisites in there? Is it really necessary for you to require Docker?

[ashishdhingra]

@Hammatt Just for reference, the error is coming as a result of the logic here. For Native AOT, there is a paragraph which mentions the requirement for Docker:

Native AOT compiles code for a specific OS version. If you run the dotnet publish command on your machine, the compiled code only runs on the OS version and processor architecture of your machine. For your application code to run in Lambda using native AOT, the code must be compiled on the [Amazon Linux 2](https://aws.amazon.com/amazon-linux-2/) (AL2) OS. The new tooling supports compiling your Lambda functions within an AL2-based Docker image, with the compiled application stored on your local hard drive.

Hope this provides some insights on why Docket is required if compiling on non-AL2 based OS.

Thanks, Ashish

[Hammatt]

@ashishdhingra Does this mean that if we use the right docker image (an AL2 based one?) it wouldn't need access to Docker?

[ashishdhingra]

@ashishdhingra Does this mean that if we use the right docker image (an AL2 based one?) it wouldn't need access to Docker?

@Hammatt Looking at the logic in the AWS .NET CLI Extensions tooling, this appears to be true. So you might give it a try and report issue, if any, here.

[normj]

@Hammatt Adding configuration to @ashishdhingra statement if the CLI tooling detects it is running in an AL2 environment like an AL2 container then it won't try spin up a container. The tough part of the dev cycle of Native AOT is the .NET code needs to compiled on the same OS as where it will run so spinning up a container was our best option when packaging on a non-AL2 environment.

[Hammatt]

Thanks, I've managed to build the lambda using the public.ecr.aws/sam/build-dotnet7:latest-x86_64 image specified in this project 👍

We're looking at building some support into our CI pipelines to support this. To help understand the best way to implement this, is this going to be the only way to run .NET 7 lambda functions, or will we also see something like a dotnet7 runtime available later on too.

Trying to decide if we should change it so that if it detects that it's a .NET 7 lambda we should build it this way, or if we should build in a separate flag into our build system to tell it weather to use this new AOT build or not.

[ashishdhingra]

Thanks, I've managed to build the lambda using the public.ecr.aws/sam/build-dotnet7:latest-x86_64 image specified in this project 👍

We're looking at building some support into our CI pipelines to support this. To help understand the best way to implement this, is this going to be the only way to run .NET 7 lambda functions, or will we also see something like a dotnet7 runtime available later on too.

Trying to decide if we should change it so that if it detects that it's a .NET 7 lambda we should build it this way, or if we should build in a separate flag into our build system to tell it weather to use this new AOT build or not.

@Hammatt Thanks for sharing your results. If no action is required from our side, please confirm if we could close this issue.

[normj]

@Hammatt To your question on will there be a managed .NET 7 runtime, Lambda's policy is to only support LTS versions for managed runtimes. So for .NET 7 the options are to deploy as a self contained or native aot function to the the provided.al2 runtime.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.