Deploying new version of lambda function

duarten commented 4 years ago

:question: General Issue

The Question

I'm attempting to setup a CodeDeploy deployment group for a Lambda function. The CDK documentation for the Version class states:

If you want to deploy through CloudFormation and use aliases, you need to add a new version (with a new name) to your Lambda every time you want to deploy an update. An alias can then refer to the newly created Version.

This suggests that if I want to make a new CodeDeploy deployment, I should change the version name. For this end, I'm naming the versions using the sha1 of the latest Git commit that affects the Lambda's code or configuration. However, if I commit code that makes cosmetic changes to the configuration (i.e., to CDK code pertaining to the lambda), that will produce no differences in the CloudFormation template, and I will get the error A version for this Lambda function exists ( 1 ). Modify the function to create a new version.

This suggests that for a new version to be deployed I need to change the code or make semantic changes to the configuration. If this is the case, why require the version name to be unique between versions?

Code:

const lambdaName = basename(__dirname)

async function getRevision(dir: string): Promise<string> {
    const root = basename((await run("git rev-parse --show-toplevel")).line())
    const p = dir.substring(dir.lastIndexOf(root) + root.length + 1, dir.length)
    return (await run(`git rev-list --abbrev-commit -1 HEAD -- ${p}`)).line()
}

export async function myLambda(stack: Stack): Promise<Alias> {
    const f = new Function(stack, lambdaName, {
        runtime: Runtime.GO_1_X,
        handler: "main",
        code: Code.asset(join(__dirname, "lambda.zip")),
    })

    const alias = new Alias(stack, lambdaName + "-alias", {
        aliasName: "live",
        version: f.addVersion(await getRevision(__dirname)),
    })

    new LambdaDeploymentGroup(stack, lambdaName + "-deployment", {
        alias: alias,
        deploymentConfig: LambdaDeploymentConfig.LINEAR_10PERCENT_EVERY_10MINUTES,
    })

    return alias
}

Environment

**CDK CLI Version: 1.18.0
**Module Version: 1.18.0
**OS: macOS Catalina
**Language: Typescript

Other information

skinny85 commented 4 years ago

Hello @duarten ,

the error you're getting:

"A version for this Lambda function exists ( 1 ). Modify the function to create a new version.".

I assume is coming up during deployment, in CloudFormation?

Thanks, Adam

duarten commented 4 years ago

Hi @skinny85,

Yes, that's correct.

skinny85 commented 4 years ago

Right. So the way we usually deal with that in the CDK is to have a new version for every synthesis - this way, the new version will always be created. Take a look at this example in our docs.

duarten commented 4 years ago

Correct me if I'm wrong, but in that case won't a synthesis always change from the previous one, even though the Lambda's code and configuration didn't? It also doesn't seem to solve my original problem, since I may deploy a stack which contains a lambda function whose code and configuration didn't change, but the CDK version did, thus leading to the "A version for this Lambda function exists ( 1 ). Modify the function to create a new version." error.

skinny85 commented 4 years ago

Correct me if I'm wrong, but in that case won't a synthesis always change from the previous one, even though the Lambda's code and configuration didn't?

Yes. It's a tradeoff: either you remember to change it, or you accept the fact that you'll get a new version every deployment. I don't think it's too big of an issue to be honest.

It also doesn't seem to solve my original problem, since I may deploy a stack which contains a lambda function whose code and configuration didn't change, but the CDK version did, thus leading to the "A version for this Lambda function exists ( 1 ). Modify the function to create a new version." error.

No. It doesn't matter that the code/configuration did not change, there will be a new version, so you won't get that error.

duarten commented 4 years ago

No. It doesn't matter that the code/configuration did not change, there will be a new version, so you won't get that error.

Sorry, I don't follow. The issue is that I specified a new version, and I got that error. If I don't change the version, then the CloudFormation template is the same and CDK doesn't attempt a deployment. If I change the version, CDK will attempt a deployment, but will fail with that error. (I guess that's some other layer complaining that the Lambda's code and configuration have not changed.)

Example:

 15/24 | 12:23:34 AM | CREATE_IN_PROGRESS   | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version86f7530 (cognitopostconfirmationnewuserVersion86f753088FD94FA)
 15/24 | 12:23:35 AM | CREATE_IN_PROGRESS   | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version86f7530 (cognitopostconfirmationnewuserVersion86f753088FD94FA) Resource creation Initiated
 16/24 | 12:23:36 AM | CREATE_COMPLETE      | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version86f7530 (cognitopostconfirmationnewuserVersion86f753088FD94FA)

Note that version is 86f7530. Changing that to 7dad47a:

 0/4 | 12:29:33 AM | CREATE_IN_PROGRESS   | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version7dad47a (cognitopostconfirmationnewuserVersion7dad47aE2212F63)
 1/4 | 12:29:33 AM | CREATE_FAILED        | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version7dad47a (cognitopostconfirmationnewuserVersion7dad47aE2212F63) A version for this Lambda function exists ( 1 ). Modify the function to create a new version.
    new Version (/Users/duarten/code/umani/bazel-bin/deploy/app.sh.runfiles/npm/node_modules/@aws-cdk/aws-lambda/lib/lambda-version.js:28:25)
    \_ Function.addVersion (/Users/duarten/code/umani/bazel-bin/deploy/app.sh.runfiles/npm/node_modules/@aws-cdk/aws-lambda/lib/function.js:259:16)
    \_ UmaniLambda.<anonymous> (/Users/duarten/code/umani/bazel-bin/deploy/app.sh.runfiles/umani/constructs/umani-lambda.js:51:37)
    \_ Generator.next (<anonymous>)
    \_ fulfilled (/Users/duarten/code/umani/bazel-bin/deploy/app.sh.runfiles/umani/constructs/umani-lambda.js:4:58)
    \_ processTicksAndRejections (internal/process/task_queues.js:93:5)

skinny85 commented 4 years ago

Hmm, you are correct. Apologies. I wonder whether Lambda added this validation recently...? I swear this used to work.

duarten commented 4 years ago

No worries :)

skinny85 commented 4 years ago

So here's my research on the topic.

Currently, we recommend customers to do the following:

    const version = func.addVersion(new Date().toISOString()); // <==
    const alias = new lambda.Alias(this, 'LambdaAlias', {
      aliasName: 'Prod',
      version,
    });

    new codedeploy.LambdaDeploymentGroup(this, 'DeploymentGroup', {
      alias,
      deploymentConfig: codedeploy.LambdaDeploymentConfig.LINEAR_10PERCENT_EVERY_1MINUTE,
    });

However, it seems that this no longer works - if the Lambda itself is unchanged from the previous version, the new Version will fail creation.

This needs some pretty serious changes in our API:

We should create a new API that substitutes Function.addVersion(), like Function.addHashedVersion() (name TBD of course). It should base the name of the version on the hash of the code property of the Function (either the asset, or the inline string) - if the code does not change, a new Version will not be created.
We should probably deprecate the old addVersion() method, because it seems like it's very easy to shoot yourself in the foot using it.

skinny85 commented 4 years ago

Actually, I figured out a workaround :) changing the description of the Function is enough to make the Version creation succeed, so this works:


        const func = new lambda.Function(this, 'lambdaName', {
            // whatever properties you need...
            description: `Generated on: ${new Date().toISOString()}`,
        });

        const version = func.addVersion(new Date().toISOString());

        const alias = new lambda.Alias(this, 'lambdaName-alias', {
            aliasName: 'live',
            version: version,
        });

        new codedeploy.LambdaDeploymentGroup(this, 'lambdaName-deployment', {
            alias: alias,
            deploymentConfig: codedeploy.LambdaDeploymentConfig.LINEAR_10PERCENT_EVERY_1MINUTE,
        });

duarten commented 4 years ago

Thanks for the workaround :)

cbertozzi commented 4 years ago

thanks for the working workaround ;) I'm facing the same issue and I tried many actions before. I think that a better way to solve this problem is to wait until the SAM CDK module is finally stable and released and then use the autoPublishAlias property that takes care for all the dynamic of recognize a new codebase and create (or not) a new version

skinny85 commented 4 years ago

I'm glad it worked @cbertozzi :) But I think the addHashedVersion() method I talked about above is the way to go here (determine the name of the function's version based on the hash of its code property), not the SAM package.

eladb commented 4 years ago

+1 on addHashedVersion. I would just make name an optional argument for addVersion and default to the asset source hash. @nija-at definitely worth getting into our planning.

hoegertn commented 4 years ago

Additionally, I would love a feature to set the version to be retained on update. Sometimes I want the new version to be deployed but old versions to be retained for callers outside of my Stack. (e.g. Alexa Skills)

kjpgit commented 4 years ago

I'm just trying to get AWS Lambda provisioned concurrency and autoscaling working, and ran into this issue. I think this is ridiculous - I don't care about versions or aliases, but I have to use them to get provisioned concurrency, but versions and aliases are broken in the CDK. Great. I suggest you guys think of an easier way to "deploy a lambda with provisioned autoscaling" with CDK.

The workaround of "random description for lambda fn" seems to work for me, but man are things slow. A single function alias update takes 2 1/2 minutes with provisioned concurrency:

aripalo commented 4 years ago

There's also the case of Lambda@Edge, which I think relates to this issue:

CloudFront requires that a specific version of Lambda function is associated with the distribution.

Currently my options are:

Manually control the Lambda versions and only create a new version when I know the Lambda code has changes. This is somewhat annoying as one tends to forget to do that.

… or

Always create a new version of the Lambda function (even if there are no changes to the actual code). This in turn will always trigger a CloudFront update which takes 20 mimutes and this is even more annoying 😅

I'd like to see exactly that kind of code hash based solution @skinny85 suggested in https://github.com/aws/aws-cdk/issues/5334#issuecomment-562979282 as it should resolve this challenge with Lambda@Edge.

stefanolczak commented 4 years ago

+1 on addHashedVersion. I would just make name an optional argument for addVersion and default to the asset source hash. @nija-at definitely worth getting into our planning.

What about other information that belongs to specified lambda's version? The function version includes also information like lambda runtime and all of the function settings, including the environment variables. Changing the settings without modifying code is possible so I think using just asset source hash as a version name is not enough. Instead we should somehow use hash of whole "AWS::Lambda::Function" resource.

aripalo commented 4 years ago

Yeah it would be nice if one could do something in the manner of:

const fn = new lambda.VersionedFunction(this, "MyFunc", { /*...*/});

… which would calculate a combined hash of the source code and of the AWS::Lambda::Function.

Then it would also create a new version if the hash has changed.

Not sure if this is doable, but just thinking out loud.

iph commented 4 years ago

+1 on addHashedVersion. I would just make name an optional argument for addVersion and default to the asset source hash. @nija-at definitely worth getting into our planning.

What about other information that belongs to specified lambda's version? The function version includes also information like lambda runtime and all of the function settings, including the environment variables. Changing the settings without modifying code is possible so I think using just asset source hash as a version name is not enough. Instead we should somehow use hash of whole "AWS::Lambda::Function" resource.

[From Lambda, just throwing opinion here]

Ran a few experiments, and this is partially dangerous and can lead to weird edge cases, though I do agree with the opinion that we should do this.

As an example, let's say you have the following environment variables in your function:

      Environment:
        Variables:
          hi: there
          hi2: there

and you "update" to:

      Environment:
        Variables:
          hi2: there
          hi: there

Lambda does not see this as an update to your function. It's an idempotent "update", because it's a map which has no explicit ordering. If you were to try to publish-version, and you were at version 1, Lambda would idempotently "publish" version 1 again.

Cloudformation does not interact kindly when you go from 1 -> 1. It will fail updates with Modify the function to create a new version. (this isn't Lambda). My guess is they are deeply tied in with the function version arn. Why? ¯\_(ツ)_/¯. I'll pop an issue in their queue, since that's a bit strange. Unfortunately, that could take some time to fix, so let's focus within the constraints of the problem presented.

If we were to naively hash the text within the function, Cloudformation will fail this update if you move from version{OLDHASH} to version{NEWHASH} in cloudformation logical id. So we will need to take extra steps to figure out what is idempotent in Lambda's eyes and produce a consistent hash with Lambda's expectations (e.g. sort a map and then produce a consistent hash off that sorted map, sort all function resource names before hashing). Tags and VPC configs are commutative, so would probably need to be sorted too. I think layer order is not commutative, but worth testing.

Sorry that it's frustratingly hard to get this right. We (Lambda+Cfn) need to do a bit better with these interactions and I'll start opening up communications to see what we can do --but due to backwards compatibility constraints of the past 5 years since this was originally shipped, the ship of "changing this" may have sailed.

iph commented 4 years ago

Examples of what I'm talking about above here: https://github.com/iph/lambda-experiments/tree/master/versioning-updates-cfn

Specifically experiment 3 in the README.

michaelfecher commented 3 years ago

Facing the same issue in combination with CDK pipelines. Are there any updates about a non-workaround solution to deploy each and every time? More critically, the .addVersion method is deprecated. In general, do you recommend using Aliases and this versioning mechanism?

dudutwizer commented 3 years ago

Facing the same issue with SAM... any ideas for workarounds?

adrian-baker commented 3 years ago

A single function alias update takes 2 1/2 minutes with provisioned concurrency:

5-6 minutes for mine . Slower than a rolling deploy of a small ECS Fargate cluster : (

iancullinane commented 2 years ago

I am using a lambda.DockerImageFunction and it seems like existing workarounds do not help.

mikoz93 commented 1 year ago

Is there any fix to this? Creating and deploying new versions on each cdk deploy is a massive waste of resource.

skinny85 commented 1 year ago

@Mikoz93 this should be fixed with hotswap deployments.

ajhool commented 1 year ago

When using hotswap deployments along with provisioned concurrency and aliases, I'm still seeing the same delay. The Alias continues to weight the old version at 100% and the new version at 0% for several minutes. I've also tried adding an AllAtOnce deployment group but that didn't work either. The deployment group appeared to have no effect on the Alias deployment and version cutover

    new LambdaDeploymentGroup(this, "DeploymentGroup", {
      alias,
      deploymentConfig: LambdaDeploymentConfig.ALL_AT_ONCE,
    });

When using hotswap without aliases, the hotswap switch is instantaneous.

skinny85 commented 1 year ago

@ajhool I think that’s because you’re using a CodeDeploy deployment group. Try removing it, and the Alias should be updated immediately.

jtaub commented 6 months ago

I know this is closed, but it's hard to tell if this issue is really resolved or not. Is it still necessary to append a timestamp to the description? As far as I can tell, it is. But if not, in what version of the CDK is this issue fixed?

skinny85 commented 6 months ago

@jtaub it should be resolved, unless you put the Alias inside a CodeDeploy DeploymentGroup.

aws / aws-cdk