search

aws / aws-extensions-for-dotnet-cli

Extensions to the dotnet CLI to simplify the process of building and publishing .NET Core applications to AWS services
Apache License 2.0
370 stars 87 forks source link

BUG: dotnet lambda package-ci creates faulty CloudFormation output template with CodeBuild #27

Closed ThomasJaeger closed 5 years ago

ThomasJaeger commented 6 years ago

I created a CloudFormation script with two dotnet core C# lambdas named:

ProcessCreateAccountFromDynamoDBEvent CreateAccount

When my buildspec.yml file is being executed in CodeBuild, it creates a wrong CloudFormation template output file where it ignores and overwrites the S3Key of Lambda ProcessCreateAccountFromDynamoDBEvent.

Attached is a comparison of the original CloudFormation stack and the output templated generated. The S3Key produced was CreateAccount-636702840090957270.zip. This file was supposed to go ONLY to the CreateAccount Lambda but instead it uses the same key for two seperate Lambdas including the ProcessCreateAccountFromDynamoDBEvent Lambda.

Please fix this ASAP. Thank you!

image

normj commented 6 years ago

The current behavior with this feature is to replace all code locations. I agree with what you are doing is valid but what I need to figure out is if I change the code to only update code locations that are blank would that cause a breaking change to anybody.

ThomasJaeger commented 6 years ago

It looks like either your regex is wrong or you simply replacing anything within a string that matches the name. It should be an exact string match and not string within a string. As it stands, I can not use CodeBuild. Is there a workaround until you fix this bug? How are other people using CodeBuild and with CloudFormation templates that contain more than one Lambda?

Should I just have empty S3Keys? But then, how do you know which S3Key should be updated if you are not doing an exact string match?

Also, are you expecting the CloudFormation template to ONLY include the Lambda function that is being build and never have more than one Lambda in the template?

Thank you for your help in advance! I appreciate it.

normj commented 6 years ago

Right now the code doesn't do any sort of filtering. The command assumes you are deploying a single .NET Core project with its dependences. That project contains 1 or more Lambda functions that are defined within a template. The template is updated to where the project was deployed. Here is the update code if you are curious.

https://github.com/aws/aws-extensions-for-dotnet-cli/blob/22b90de930c2f12fb58117b426d054990e2b82b3/src/Amazon.Lambda.Tools/LambdaUtilities.cs#L384

What I'm confused about is where ProcessCreateAccountFromDynamoDBEvent is defined and how is that project deployed. It seems like you are saying that that function is not in the same project as CreateAccount.

If they are defined in 2 separate unrelated projects then you need to either deploy them separately or use the dotnet lambda package command to create just the deployment packages for both projects and then use the aws cli command aws cloudformation package with the template pointing to the local location of the zip files created by dotnet lambda package.

The dotnet lambda commands are currently project specific right now as that is how tools worked in pre .NET Core 2.1. We are working on converting these project level commands into a new .NET Core 2.1 global tools which would make scenarios across projects more feasible. This change won't be in the first version of the global tools version but it will gives us a path to make more cross project scenarios easier with the dotnet lambda commands.

ThomasJaeger commented 6 years ago

Thank you for your feedback, Norm. I originally started the CloudFormation template manually with two Lambda projects. The ProcessCreateAccountFromDynamoDBEvent project was the second one I had created. However, I'm trying to make this all work with CodeCommit, CodeBuild and CodeDeploy and eventually CodePipeline.

I still have not decided which is better, a bunch of seperate CloudFormation scripts (one for each Lambda and all its dependencies), or just one CloudFormation script. The problem I see is that many Lambdas will be sharing many services like API Gateway, DynamoDB, etc. It seems that if I split-up my CloudFormation script into many, I'm not sure yet how I would hanlde regular updates to shared code that many Lambdas use such as my domain model in domain driven design (DDD). If I update one Lambda, I now have to update all other Lambdas with their seperate CloudFormation scipts just because I made a domain model change.

You see, I'm considering Lambdas just the entry points into the system and the core domain actually doing the intelligent work (based on a CQRS and EventSource model).

Do you have any suggesstions or recommendations? Thank you in advance!

normj commented 6 years ago

I just pushed out version 3.1.0 of Amazon.Lambda.Tools which I think will solve your problem. The version reworks how the template is parsed to look to see where each Lambda function in the template code property is pointing to and update the template based on that. In your case the Lambda functions that already have an S3Bucket field in will assume the code has already been uploaded and leave them alone.

If S3 bucket was empty it would look for the zip file locally and if found upload to S3 and update the location in the template. You can also have each Lambda function point to different .NET Core projects and each one will be built and uploaded individually. If the S3 key is empty or missing than the code assumes you want to upload the current directory for the code of the Lambda function.

ThomasJaeger commented 5 years ago

Thank you, Norm. Appreciate all your efforts!

normj commented 5 years ago

Thanks!