Could not deploy lambda the first time

[eyal-mor]

While deploying a Lambda function for the first time the stage may fail when executing the task - Lambda Event Configuration Task.

Error:

Exception ( Lambda Event Configuration Task )

Could not find lambda to update event config for

This task is executed regardless of if an event source was set or not.

Reproducing

1. Create a Lambda Create stage
2. Save Pipeline
3. Execute Pipeline

This issue is intermittent, and no root cause was identified in the AWS environment.

The second run of the pipeline will pass successfully.

Reference JSON used to create/execute the stage:

{
    "account": "${parameters[\"account\"]}",
    "batchsize": 10,
    "cloudProvider": "aws",
    "deadLetterConfig": {
      "targetArn": ""
    },
    "detailName": "",
    "enableLambdaAtEdge": false,
    "envVariables": {},
    "functionName": "${execution.application}-${parameters[\"lambda_function_name\"]}",
    "functionUid": "${parameters[\"lambda_function_name\"]}",
    "handler": "${parameters[\"lambda_handler\"]}",
    "memorySize": 128,
    "name": "Deploy Spinnaker Events Lambda",
    "publish": true,
    "region": "${parameters[\"aws_region\"]}",
    "reservedConcurrentExecutions": 10,
    "role": "${parameters[\"role_arn\"]}",
    "runtime": "${parameters[\"lambda_runtime\"]}",
    "s3bucket": "${parameters[\"s3_bucket\"]}",
    "s3key": "${parameters[\"s3_key\"]}",
    "securityGroupIds": [],
    "stackName": "${parameters[\"environment\"]}",
    "subnetIds": [],
    "tags": {
      "purpose": "Lambda Deployment"
    },
    "timeout": 90,
    "tracingConfig": {
      "mode": "PassThrough"
    },
    "triggerArns": [],
    "type": "Aws.LambdaDeploymentStage"
  }

[shaiguitar]

I'm running into this. Found this related issue: https://github.com/spinnaker-plugins/aws-lambda-deployment-plugin-spinnaker/issues/51 - not sure if the root cause is related or the error message is the commonality. It could be related to timing or a race condition of caching or saving the data in clouddriver vs creation/querying the upstream resource.

[coleduclos]

@shaiguitar and I have been able to consistently reproduce this issue using version 1.0.6 of this plugin while deploying functions to us-east-1. However, unlike mentioned above, a second execution of the pipeline will continue to fail with Function already exist: <function name> (Service: AWSLambda; Status Code: 409. (Seems related to #73 )

Likely Cause

It is taking much longer to create a function in us-east-1 and the Lambda Event Configuration Task is timing out. The error is thrown at L70 of LambdaUpdateEventConfigurationTask.java:

    public TaskResult execute(@NotNull StageExecution stage) {
       ...
        LambdaDefinition lf = utils.findLambda(stage, justCreated);
        if (lf == null) {
            return formErrorTaskResult(stage, String.format("Could not find lambda to update event config for"));
        }
       ...
    }

This means that utils.findLambda cannot find the Lambda Function after creating it, even though it attempts to find the function 5 times (link to code here):

    public LambdaDefinition findLambda(StageExecution stage, boolean shouldRetry) {
        LambdaGetInput lgi = this.getInput(stage, LambdaGetInput.class);
        lgi.setAppName(stage.getExecution().getApplication());
        //LambdaGetOutput lf = (LambdaGetOutput)stage.getContext().get(LambdaStageConstants.lambdaObjectKey);
        LambdaDefinition lf = this.retrieveLambda(lgi);
        int count = 0;
        while (lf == null && count < 5 && shouldRetry == true) {
            count++;
            lf = this.retrieveLambda((lgi));
            this.await();
        }
        return lf;
    }

NOTE: We do not see this issue while deploying to us-west-2. However, we do see it occurring while deploying to us-east-1.

Short Term Solution Proposals

• Increase the number of attempts to retrieve the lambda information
• Add a longer wait time in between each attempt

Reproducing

To reproduce follow the steps below:

Create a Lambda Create stage with JSON mentioned below (notice region set to us-east-1)
Save Pipeline
Execute Pipeline

Reference JSON for Spinnaker stage (notice region set to us-east-1)

{
  "account": "${parameters[\"account\"]}",
  "batchsize": 10,
  "cloudProvider": "aws",
  "deadLetterConfig": {
    "targetArn": ""
  },
  "detailName": "",
  "enableLambdaAtEdge": false,
  "envVariables": {},
  "functionName": "${execution.application}-${parameters[\"lambda_function_name\"]}",
  "functionUid": "${parameters[\"lambda_function_name\"]}",
  "handler": "${parameters[\"lambda_handler\"]}",
  "memorySize": 128,
  "name": "Deploy Spinnaker Events Lambda",
  "publish": true,
  "region": "us-east-1",
  "reservedConcurrentExecutions": 10,
  "role": "${parameters[\"lambda_role_arn\"]}",
  "runtime": "${parameters[\"lambda_runtime\"]}",
  "s3bucket": "${parameters[\"s3_bucket_name\"]}",
  "s3key": "${parameters[\"s3_key\"]}",
  "securityGroupIds": [],
  "stackName": "${parameters[\"environment\"]}",
  "subnetIds": [],
  "tags": {
    "purpose": "Lambda Deployment",
  },
  "timeout": 5,
  "tracingConfig": {
    "mode": "PassThrough"
  },
  "triggerArns": [],
  "type": "Aws.LambdaDeploymentStage"
}

[nimakaviani]

Great details, thanks @coleduclos ... I have been using us-west-2 primarily. possibly why this appeared to be a non-issue. will try with us-east as well. should be an easy fix.