Unable to use dotnet lambda deploy-function in Linux for AOT

[jasonterando]

Describe the bug

When executing dotnet lambda deploy-function in Linux as a non-root user, it executes Docker, which creates directories using the root account. Subsequently the .zip command fails (outside of the Docker container) because the current user does not have write to the bin/Release folder, which was created by the Docker's root user.

As a workaround, I could install dotnet and execute builds as root, but for security considerations, this isn't ideal.

Expected Behavior

dotnet lambda deploy-function should run successfully without error as a non-root user

Current Behavior

Trail of tears:

$ dotnet lambda deploy-function
Amazon Lambda Tools for .NET Core applications (5.6.2)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Found /etc/os-release
Architecture not provided, defaulting to x86_64 for container build image.
Executing publish command
Starting container for native AOT build using build image: public.ecr.aws/sam/build-dotnet7:latest-x86_64.
... invoking 'docker run --rm --volume "/tmp/Temp-e74a018c-de14-4796-ac6e-aa23bbed068e/LambdaNativeAot/src/LambdaNativeAot":/tmp/source/ --name tempLambdaBuildContainer-6c5d3da2-44a5-49e2-81d5-696f07f3a64a -i public.ecr.aws/sam/build-dotnet7:latest-x86_64 dotnet publish "/tmp/source/" --output "/tmp/source/bin/Release/net7.0/publish" --configuration "Release" --framework "net7.0" --self-contained true /p:GenerateRuntimeConfigurationFiles=true --runtime linux-x64 /p:StripSymbols=true' from directory /tmp/Temp-e74a018c-de14-4796-ac6e-aa23bbed068e/LambdaNativeAot/src/LambdaNativeAot
... docker run: MSBuild version 17.4.0+18d5aef85 for .NET
... docker run:   Determining projects to restore...
... docker run:   Restored /tmp/source/LambdaNativeAot.csproj (in 20.21 sec).
... docker run:   LambdaNativeAot -> /tmp/source/bin/Release/net7.0/linux-x64/bootstrap.dll
... docker run:   LambdaNativeAot -> /tmp/source/bin/Release/net7.0/publish/
Missing deps.json file. Skipping flattening runtime folder because  is an unrecognized format
Zipping publish folder /tmp/Temp-e74a018c-de14-4796-ac6e-aa23bbed068e/LambdaNativeAot/src/LambdaNativeAot/bin/Release/net7.0/publish to /tmp/Temp-e74a018c-de14-4796-ac6e-aa23bbed068e/LambdaNativeAot/src/LambdaNativeAot/bin/Release/net7.0/LambdaNativeAot.zip
... zipping: zip I/O error: Permission denied
... zipping: zip error: Could not create output file (/tmp/Temp-e74a018c-de14-4796-ac6e-aa23bbed068e/LambdaNativeAot/src/LambdaNativeAot/bin/Release/net7.0/LambdaNativeAot.zip)
Unknown error executing command: Could not find file '/tmp/Temp-e74a018c-de14-4796-ac6e-aa23bbed068e/LambdaNativeAot/src/LambdaNativeAot/bin/Release/net7.0/LambdaNativeAot.zip'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
   at System.IO.File.ReadAllBytes(String path)
   at Amazon.Lambda.Tools.Commands.DeployFunctionCommand.PerformActionAsync() in C:\build\src\Amazon.Lambda.Tools\Commands\DeployFunctionCommand.cs:line 276
   at Amazon.Common.DotNetCli.Tools.Commands.BaseCommand`1.ExecuteAsync() in C:\build\src\Amazon.Common.DotNetCli.Tools\Commands\BaseCommand.cs:line 47

Reproduction Steps

In Linux, logged in as a non-root user:

dotnet new lambda.NativeAOT -n LambdaNativeAot
cd LambdaNativeAot/src/LambdaNativeAot/
dotnet lambda deploy-function

Possible Solution

On Linux (and maybe MacOS?) use id to get the current user and group IDs, and launch Docker using the -u and -g parameters containing these IDs (other changes may have to be made to the Docker image to allow access to directories like ./dotnet)

Additional Information/Context

No response

Targeted .NET platform

.NET 7

CLI extension version

Package Id                                     Version         Commands                 
----------------------------------------------------------------------------------------
amazon.lambda.tools                            5.6.2           dotnet-lambda            
aws.codeartifact.nuget.credentialprovider      1.0.0           dotnet-codeartifact-creds
coverlet.console                               3.1.2           coverlet                 
dotnet-counters                                5.0.251802      dotnet-counters          
dotnet-ef                                      6.0.4           dotnet-ef                
dotnet-reportgenerator-globaltool              5.1.10          reportgenerator          
dotnet-svcutil                                 2.0.3           dotnet-svcutil           
dotnet-trace                                   5.0.251802      dotnet-trace             
swashbuckle.aspnetcore.cli                     6.2.3           swagger      

Environment details (OS name and version, etc.)

Linux (Mint, Ubuntu)

[ashishdhingra]

Hi @jasonterando,

Good afternoon.

I'm unsure how AWS .NET Extensions CLI tooling would handle that scenario since it delegates the process to Docker daemon, which by default runs with the root privilege. I came across article Run Docker as a non-root user that proposes some workaround to mitigate your scenario. Please check. Kindly test the steps in a non-prod environment, AWS team is not responsible for any side-effects to the system, the referenced article is just provided for guidance purposes.

Thanks, Ashish

[jasonterando]

Hi, @ashishdhingra, I took a look at the article and, unfortunately, it doesn't really provide a mechanism to override the default user when launching a Docker run, it still has to be specified on the CLI when executing the run.

There ends up being another problem, as well. By default, the .NET Docker container builds out .NET and NuGet in the /.dotnet and /.local. This also ends up breaking when running as non-root. Fortunately, somebody figured out how to get around this by setting environment variables, see GitHub dotnet issue 7868

So, I've taken a stab at fixing this. I'll outline the approach below, along with a link to a GitHub fork of aws-extensions-for-dotnet-cli that implements the approach. If this is something that you guys think is worth including, I can submit a pull request. The footprint of all updates is in the Amazon.Common.DotNetCli.Tools namespace. Nothing should change for Windows users.

1. Determine if dotnet.lambda is being run in Linux or MacOS.
2. If so, then get the user's UID and GID using the id command available in Linux and MacOS using System.DiagnosticsProcess. Inject these values using "-u UID:GID" in the generated Docker run command (DockerCLIWrapper).
3. Also, set environment variables to set the .NET and NuGet folders to be within /tmp

The fork is here. The update to Run in DockerCLIWrapper .cs starts around line 165. The code to get the Unix/MacOS user is in PosixUserHelper.cs.

Given that this is a CLI executable as opposed to a reusable library, I made PosixUserHelper a static class. If your coding standards need this refactored to a non-static class, implementing an interface, etc. I can do that. I also took the liberty of implementing a List to build the list of arguments to send to Docker.

I realize in the giant list of things you guys have on your list, this is probably trivial, so if this is not something you can do near-term, no problem, I can switch to Windows for development. My guess is that eventually, though, you'll probably end up needing to do something like this, especially if people are leveraging this tool in CI/CD pipelines where they don't want to run Docker as root.

[jasonterando]

Hi, bump-ing this. I'd like to continue using Linux for development, and I would be surprised if I'm the only one running into this. If I do a PR for this, would somebody be able to take a look? Thanks

[Beau-Gosse-dev]

Hey @jasonterando ! Sorry, I've been meaning to look into this and reproduce it myself, but have been busy with other things. I will definitely look at a PR if you can provide one, we would appreciate that very much! Although, I'm not actually an owner of this repo and wouldn't be able to merge it myself, but can try to find the right person. It definitely seems like an issue that we want to fix.

[normj]

Closing now that PR has been released as part of version 5.6.3 of Amazon.Lambda.Tools

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[jasonterando]

Thanks for shepherding this through

[normj]

Thanks for your contribution!