aws-lambda: ARM lambdas resulting in bad-format

[givsly-stephen]

Describe the bug

Deploying a ARM lambda using Pipelines (in-series) works on DEV and TEST but FAILS on PROD

Expected Behavior

I would expect in a serial deployment from DEV -> TEST -> PROD to see some consistency.

This troubles me especially because I rely on DEV and TEST to ensure what I push to PROD worked fine.

Current Behavior

I approve for PROD, and it deploys, but the LAMBDA is DEAD. Gateway returns

{
    "message": "Internal Server Error"
}

In cloudwatch I see

Error: fork/exec /opt/extensions/cloudwatch_lambda_agent: exec format error
Extension.LaunchError

Reproduction Steps

I have a golang lambda

    const goLambda = new Function(this, 'V1FiberCampaignApi', {
      code: Code.fromAsset(path.join(__dirname, '../../out/lambda.zip')),
      handler: 'bootstrap',
      architecture: Architecture.ARM_64,
      runtime: Runtime.PROVIDED_AL2,

which is build in a Makefile using

    GOOS=linux CGO_ENABLED=0 GOARCH=arm64 go build -tags lambda.norpc -o $(OUT_DIR)/bootstrap

All this is deployed using pipelines in series: DEV -> TEST -> PROD

    cdkPipeline.addStage(devState);
    cdkPipeline.addStage(testStage, testOptions);
    cdkPipeline.addStage(prodStage, prodOptions);

Now, it all DEPLOYES without build issues, or any indication of issues.

When testing the LAMBDA on DEV then TEST all PASS and the REST API is accessible.

I approve for PROD, and it deploys, but the LAMBDA is DEAD. Gateway returns

{
    "message": "Internal Server Error"
}

In cloudwatch I see

Error: fork/exec /opt/extensions/cloudwatch_lambda_agent: exec format error
Extension.LaunchError

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.100

Framework Version

No response

Node.js Version

18

OS

linux

Language

Go

Language Version

1.20.10

Other information

No response

[peterwoodworth]

prod is just a name, right? If you're having issues just in the prod environment wouldn't that be an issue with how you've defined prod + props? The details aren't present in how you've defined these environments differently

[givsly-stephen]

Yes, prod is our name for production.

But apart from

    const devState = new EnvStage(this, DevEnvironment);
    const testStage = new EnvStage(this, TestEnvironment);
    const prodStage = new EnvStage(this, ProdEnvironment);

    const testOptions: AddStageOpts = {
      pre: [
        new ManualApprovalStep('ApproveTest', {
          comment: 'Deploy this version to test?',
        }),
      ],
      post: [],
    };

    const prodOptions = {
      pre: [
        new ManualApprovalStep('ApproveProd', {
          comment: 'Deploy this version to production?',
        }),
      ],
    };

    cdkPipeline.addStage(devState);
    cdkPipeline.addStage(testStage, testOptions);
    cdkPipeline.addStage(prodStage, prodOptions);

But the const cdkPipeline = new CodePipeline() is identical across all stages.

So technically what is delivered to all DEV -> TEST -> PRODUCTION is the same code branch (main)

I cannot see how it differs.

ALSO I have 3 other repos we use the exactly same method and it does work fine ARM on PROD.

Scratching my head.

[jamestelfer]

I'm not sure if the issue, but the error is not pointing at your executable, but the cloud watch extension:

opt/extensions/cloudwatch_lambda_agent

If this is part of your zip, double check what you're including.

If the above executable is part of the standard al2 environment, talk to AWS support.

Also, try synthesising each environment and comparing the generated cloud assemblies, this might show an inadvertent difference that is otherwise not apparent.

If this seems infeasible, compare the template from the cloudformation stacks.

[givsly-stephen]

I will open a ticket with AWS Support.

Regarding the above, I do not include any custom extensions and certainly not cloudwatch_lambda_agent. My golang zip only includes one binary called bootstrap as needed per the AWS instructions, and this worked fine on our DEV and TEST aws accounts.

Thanks for your inputs,

[jamestelfer]

I had a quick look just for interest's sake:

• This extension is used by Cloudwatch Lambda Insights
• You enable this in CDK using the FunctionProps.insightsVersion property
• this can also be enabled manually in the console
• when activated, it adds a layer to your Lambda that includes this extension executable.
• it does support ARM, but regional availability varies, check the doco

[givsly-stephen]

I am very grateful you took another look, i would not have seen this one.

it adds a layer to your Lambda that includes this extension executable.

Brilliant - this explains that. Activated from the UI, not the code.

Fishy how this was implemented, and i am sure it should hit others too.

Shouldn't switching to ARM kinda invalidate this extension and re-installed somehow automatically??

Is this something you will push upwards to AWS?

Thank you and have a super weekend,

Stephen

[jamestelfer]

I'm not an AWS employee, just an interested bystander 😄

I suspect that if you've just switched this Lambda to ARM and updated by CFN, the layer added manually by the console hasn't changed from the x86 version. If you want insights, you'll probably have more success if you add it via the insightsVersion property and it will handle switching to the appropriate layer by architecture.

Depending on your cost sensitivity, this could be implemented on all environments or just in your production stage.

[givsly-stephen]

Closing the issue after confirming that disabling the insights setting from UI prior to installing the NEW format version worked fine.

Really want to thank @jamestelfer for digging into this

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.