search

aws / aws-extensions-for-dotnet-cli

Extensions to the dotnet CLI to simplify the process of building and publishing .NET Core applications to AWS services
Apache License 2.0
369 stars 86 forks source link

"lambda deploy-serverless" command tries to build image when ImageUri is set without any Metadata #264

Closed roncrush closed 1 year ago

roncrush commented 1 year ago

Describe the bug

According to the docs for "AWS::Serverless::Function" the "ImageUri" property can be set if the image has already been pushed to somewhere like ECR (https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html#sam-function-imageuri).

The behavior I'm seeing with the command "lambda deploy-serverless" is that it treats the "ImageUri" like a local path as if it's trying to build the image and push it.

Expected Behavior

The command should result in the function being successfully deployed by the CloudFormation stack.

Current Behavior

The following error appears:

Processing CloudFormation resource \<Function Name> Initiate packaging of \<ImageUri> for resource Directory that the field \<Function Name>/CodeUri-Or-ImageUri is pointing doesn't exist

Reproduction Steps

  1. Initialize the hello world sample: 
    sam init --package-type Image --base-image amazon/dotnet6-base
  2. Manually build and deploy the Dockerfile to somewhere like ECR.
  3. Update the template.yaml file to look like this (replace the \<ImageUri> with the deployed image URI): 
    
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Template to test bug with ImageUri

Globals: Function: Timeout: 10 MemorySize: 128

Resources: HelloWorldFunction: Type: AWS::Serverless::Function Properties: PackageType: Image ImageUri: 

4, Run the "dotnet lambda deploy-serverless" command (will need a S3 bucket):

dotnet lambda deploy-serverless --s3-bucket "some-bucket-for-testing-bug" --s3-prefix "bug/" --stack-name "test-bug" --template "template.yaml"



### Possible Solution

It seems like the packaging step should be skipped for Lambda Functions that have an ImageUri set without any metadata.

### Additional Information/Context

_No response_

### Targeted .NET platform

6.0.302

### CLI extension version

| Package Id           |                 Version    |  Commands |
|-------------------|---------------------|----------------------|
| amazon.lambda.tools     |              5.6.3  |      dotnet-lambda |
| dotnet-ef                         |             6.0.8   |     dotnet-ef |

### Environment details (OS name and version, etc.)

macOS Monterey Version 12.6
ashishdhingra commented 1 year ago

@roncrush Thanks for opening the issue. I think you are affected feature request https://github.com/aws/aws-extensions-for-dotnet-cli/issues/196. Could you please check and confirm?

Thanks, Ashish

roncrush commented 1 year ago

@ashishdhingra Thanks for the quick response. We actually recently updated the Amazon Lambda Tools package as part of our move from Windows to MacOS. The old version we were using was 4.3.0 and this case was working then, which is why I've reported it as a bug.

ashishdhingra commented 1 year ago

Hi @roncrush,

Good afternoon.

Issue appears to be reproducible. Below are the steps followed:

More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst

Globals: Function: Timeout: 10 MemorySize: 128

Resources: HelloWorldFunction: Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction Properties: PackageType: Image ImageUri: <>.dkr.ecr.us-east-2.amazonaws.com/helloworldtest:latest Events: HelloWorld: Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api Properties: Path: /hello Method: get Metadata: DockerTag: dotnet6-v1 DockerContext: ./src/HelloWorld Dockerfile: Dockerfile DockerBuildArgs: SAM_BUILD_MODE: run

Outputs:

ServerlessRestApi is an implicit API created out of Events key under Serverless::Function

Find out more about other implicit resources you can reference within SAM

https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api

HelloWorldApi: Description: "API Gateway endpoint URL for Prod stage for Hello World function" Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/" HelloWorldFunction: Description: "Hello World Lambda Function ARN" Value: !GetAtt HelloWorldFunction.Arn HelloWorldFunctionIamRole: Description: "Implicit IAM Role created for Hello World function" Value: !GetAtt HelloWorldFunctionRole.Arn

- Executed the command `dotnet lambda deploy-serverless --s3-bucket some-s3-bucket --s3-prefix "bug/" --stack-name "test-bug" --template ../../template.yaml`

Amazon Lambda Tools for .NET Core applications (5.6.3) Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Processing CloudFormation resource HelloWorldFunction Initiate packaging of ./src/HelloWorld for resource HelloWorldFunction Building Docker image for /Users/<>/dev/repros/samnetcliextensions-imageuri-issue264/HelloWorld/src/HelloWorld/../.././src/HelloWorld Inspecting Dockerfile to figure how to build project and docker image Executing docker build ... invoking 'docker build', working folder '/Users/<>/dev/repros/samnetcliextensions-imageuri-issue264/HelloWorld/src/HelloWorld/../.././src/HelloWorld, docker file /Users/<>/dev/repros/samnetcliextensions-imageuri-issue264/HelloWorld/src/HelloWorld/../.././src/HelloWorld/Dockerfile, image name dotnet6-v1:latest' ... docker build -f "/Users/<>/dev/repros/samnetcliextensions-imageuri-issue264/HelloWorld/src/HelloWorld/../.././src/HelloWorld/Dockerfile" -t dotnet6-v1:latest --build-arg SAM_BUILD_MODE=run . ... docker build: #1 [internal] load build definition from Dockerfile ... docker build: #1 sha256:2ffc30f135a36a692885f3525ed911bf011a8ea1565aae75efc15224d23d49a1 ... docker build: #1 transferring dockerfile: 37B done ... docker build: #1 DONE 0.0s ... docker build: #2 [internal] load .dockerignore ... docker build: #2 sha256:09532619e16bdbab5a4bbd930734381bd6248ed9e1d816ef46ccee2c46a90683 ... docker build: #2 transferring context: 2B done ... docker build: #2 DONE 0.0s ... docker build: #4 [internal] load metadata for mcr.microsoft.com/dotnet/sdk:6.0 ... docker build: #4 sha256:9eb4f6c3944cfcbfe18b9f1a753c769fc35341309a8d4a21f8937f47e94c712b ... docker build: #4 DONE 0.1s ... docker build: #3 [internal] load metadata for public.ecr.aws/lambda/dotnet:6 ... docker build: #3 sha256:b0e709de1ce280b90bfd00976d19b961170e8b446b69d5ffe1f8022f8945df19 ... docker build: #3 DONE 0.5s ... docker build: #5 [stage-1 1/2] FROM public.ecr.aws/lambda/dotnet:6@sha256:df3a25efae0d73cf367d07dfd14567adb61e2cb077177dc4dbb182cb91d2d9de ... docker build: #5 sha256:42584e7b18e9ca68e519c26d812ffb4aa20cd3a0ec5508208f92c9713a3e1fc8 ... docker build: #5 DONE 0.0s ... docker build: #6 [build-image 1/9] FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:8189c53e1f0323b8beb5fa174853f413364a203f32b0b7c2488b534a3c1e76c6 ... docker build: #6 sha256:87cb5ae0e817750b8601520c8fa2e443a0c6caeaf5c8fe8f7239ed7d5fae7993 ... docker build: #6 DONE 0.0s ... docker build: #10 [internal] load build context ... docker build: #10 sha256:ed33d67c542c3e9f175f97682f689484c760c26e448b0209906a47c7d1d1292f ... docker build: #10 transferring context: 120B done ... docker build: #10 DONE 0.0s ... docker build: #8 [build-image 3/9] RUN mkdir /build ... docker build: #8 sha256:acdf1de403df574c2f6164764641fcb98d11d56c10185fe5668d7f8966d328ea ... docker build: #8 CACHED ... docker build: #11 [build-image 5/9] COPY Function.cs HelloWorld.csproj aws-lambda-tools-defaults.json /build/ ... docker build: #11 sha256:a380f3f3bf4204e6498eafa8ed0ebc119cc5fb74f600c16856020e4fce965740 ... docker build: #11 CACHED ... docker build: #9 [build-image 4/9] WORKDIR /build ... docker build: #9 sha256:635d0c7915105740c7f02872beef8b35cc5fc46faa1d4718f31c108bb49bcf34 ... docker build: #9 CACHED ... docker build: #12 [build-image 6/9] RUN dotnet tool install -g Amazon.Lambda.Tools ... docker build: #12 sha256:9a143a9620bfd1d7049d00324ea642647cb5c2b7c1a91e93afee0b2d74c54236 ... docker build: #12 CACHED ... docker build: #14 [build-image 8/9] RUN if [ "run" = "debug" ]; then dotnet lambda package --configuration Debug; else dotnet lambda package --configuration Release; fi ... docker build: #14 sha256:c3fe948bebdc5a87fe4a7cfc9f1d5c2bf12439f76c243f059a9d8c0be5e2a0cc ... docker build: #14 CACHED ... docker build: #15 [build-image 9/9] RUN if [ "run" = "debug" ]; then cp -r /build/bin/Debug/net6.0/publish/ /build/build_artifacts; else cp -r /build/bin/Release/net6.0/publish/ /build/build_artifacts; fi ... docker build: #15 sha256:8ddf33c6341811111b805c6883bd30175a76483f97165cb0f158df6464462083 ... docker build: #15 CACHED ... docker build: #7 [build-image 2/9] RUN apt-get update && apt-get -y install zip ... docker build: #7 sha256:0e5275dcf76fc26841bd7c93fbaa32f3e86953af6086f4ff31b72e7c438f4176 ... docker build: #7 CACHED ... docker build: #13 [build-image 7/9] RUN mkdir -p build_artifacts ... docker build: #13 sha256:3ca43a04992fa5a40d2306528e90ec99b3c50be2c58b7cccaa9d707327a034ec ... docker build: #13 CACHED ... docker build: #16 [stage-1 2/2] COPY --from=build-image /build/build_artifacts/ /var/task/ ... docker build: #16 sha256:3ac39db0e8432e75f6a5e22e79b7bd2599a63fc83bf3b20463ce3606e7bdb93e ... docker build: #16 CACHED ... docker build: #17 exporting to image ... docker build: #17 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00 ... docker build: #17 exporting layers done ... docker build: #17 writing image sha256:0187e66b8f20e4127bc26ea55bfa7002272fe140c0a4080179a31aa1d1599ccc done ... docker build: #17 naming to docker.io/library/dotnet6-v1:latest done ... docker build: #17 DONE 0.0s Fetching ECR authorization token to use to login with the docker CLI Executing docker CLI login command ... invoking 'docker login' ... docker login: WARNING! Using --password via the CLI is insecure. Use --password-stdin. ... docker login: Login Succeeded Creating ECR Repository dotnet6-v1 Taging image dotnet6-v1:latest with dotnet6-v1:helloworldfunction-0187e66b8f20-latest ... invoking 'docker tag' Pushing image to ECR repository ... invoking 'docker push' The push refers to repository [<>.dkr.ecr.us-east-2.amazonaws.com/dotnet6-v1] 600dc3d98bcf: Pushed 9f47e18cd643: Pushed 6f049e498202: Pushed 2d244e0816c6: Pushed c89552e947a7: Pushed c0d1687ac68f: Pushed 0d91c86bcea1: Pushed helloworldfunction-0187e66b8f20-latest: digest: sha256:84c208d02d121807ee5c988355258813e732412ce22e30e1795e2197740d6696 size: 1788 Image <>.dkr.ecr.us-east-2.amazonaws.com/dotnet6-v1:helloworldfunction-0187e66b8f20-latest Push Complete. Uploading to S3. (Bucket: some-s3-bucket Key: bug/test-bug-template-638144939575530990.yaml) ... Progress: 100% Found existing stack: False CloudFormation change set created ... Waiting for change set to be reviewed Created CloudFormation stack test-bug

Timestamp Logical Resource Id Status

3/15/2023 4:19 PM test-bug CREATE_IN_PROGRESS
3/15/2023 4:19 PM HelloWorldFunctionRole CREATE_IN_PROGRESS
3/15/2023 4:19 PM HelloWorldFunctionRole CREATE_IN_PROGRESS
3/15/2023 4:19 PM HelloWorldFunctionRole CREATE_COMPLETE
3/15/2023 4:19 PM HelloWorldFunction CREATE_IN_PROGRESS
3/15/2023 4:19 PM HelloWorldFunction CREATE_IN_PROGRESS
3/15/2023 4:19 PM HelloWorldFunction CREATE_COMPLETE
3/15/2023 4:20 PM ServerlessRestApi CREATE_IN_PROGRESS
3/15/2023 4:20 PM ServerlessRestApi CREATE_IN_PROGRESS
3/15/2023 4:20 PM ServerlessRestApi CREATE_COMPLETE
3/15/2023 4:20 PM HelloWorldFunctionHelloWorldPermissionProd CREATE_IN_PROGRESS
3/15/2023 4:20 PM ServerlessRestApiDeployment47fc2d5f9d CREATE_IN_PROGRESS
3/15/2023 4:20 PM HelloWorldFunctionHelloWorldPermissionProd CREATE_IN_PROGRESS
3/15/2023 4:20 PM ServerlessRestApiDeployment47fc2d5f9d CREATE_IN_PROGRESS
3/15/2023 4:20 PM ServerlessRestApiDeployment47fc2d5f9d CREATE_COMPLETE
3/15/2023 4:20 PM ServerlessRestApiProdStage CREATE_IN_PROGRESS
3/15/2023 4:20 PM ServerlessRestApiProdStage CREATE_IN_PROGRESS
3/15/2023 4:20 PM ServerlessRestApiProdStage CREATE_COMPLETE
3/15/2023 4:20 PM HelloWorldFunctionHelloWorldPermissionProd CREATE_COMPLETE
3/15/2023 4:20 PM test-bug CREATE_COMPLETE
Stack finished updating with status: CREATE_COMPLETE

Output Name Value

HelloWorldFunctionIamRole arn:aws:iam::<>:role/test-bug-HelloWorldFunctionRole-ZJ7T05ZPLES1 HelloWorldApi https://uli61kpfk2.execute-api.us-east-2.amazonaws.com/Prod/hello/ HelloWorldFunction arn:aws:lambda:us-east-2:<>:function:test-bug-HelloWorldFunction-4oOLPbRsRYtK


As you could see from the output above, I didn't got the error reported by you. It ignored `ImageUri` property all together and pushed image to a new ECR repository. The reason is that per https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html#sam-function-imageuri, `Building your application with necessary Metadata entries takes precedence over ImageUri, so if you specify both then ImageUri is ignored.`.

Thereafter, I removed `Metadata` entries and re-executed `dotnet lambda deploy-serverless --s3-bucket some-s3-bucket --s3-prefix "bug/" --stack-name "test-bug" --template ../../template.yaml` command. This gave the error reported by you:

Amazon Lambda Tools for .NET Core applications (5.6.3) Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Processing CloudFormation resource HelloWorldFunction Initiate packaging of <>.dkr.ecr.us-east-2.amazonaws.com/helloworldtest:latest for resource HelloWorldFunction Directory that the field HelloWorldFunction/CodeUri-Or-ImageUri is pointing doesn't exist



I think the related open feature request https://github.com/aws/aws-extensions-for-dotnet-cli/issues/196 does not document/validates the absence of `Metadata` properties (since it was reported as a feature request), hence is duplicate of this issue.

I would validate if this worked on [Amazon.Lambda.Tools](https://www.nuget.org/packages/Amazon.Lambda.Tools/)) version `4.3.0`. If it does, then we would classify this as a bug instead of a feature request and take it from there.

Thanks,
Ashish
roncrush commented 1 year ago

Hey @ashishdhingra

As a temporary fix we ended up downgrading back to 4.3.0 and it started deploying properly. I didn't test with the template.yaml you've shown, but it did work with the template.yaml that's on the initial post. When I updated to the latest version of the tool it stopped working again.

ashishdhingra commented 1 year ago

I would validate if this worked on Amazon.Lambda.Tools) version 4.3.0. If it does, then we would classify this as a bug instead of a feature request and take it from there.

This appears to work in version 4.3.0 of Amazon.Lambda.Tools. Closing other open feature request https://github.com/aws/aws-extensions-for-dotnet-cli/issues/196 to track it as part of this issue.

ashishdhingra commented 1 year ago

The fix has been released as part of Amazon.Lambda.Tools version 5.6.5

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.