Closed eboily closed 2 weeks ago
@eboily Good afternoon. Thanks for opening the issue. Somehow, I'm unable to reproduce the issue. Below is CDK stack used with associated Lambda function and layers. Used CDK version 2.161.1 (build 0a606c9)
.
src/lambdas/lambda.ts
import * as logic from '/opt/business-logic';
import * as security from '/opt/security-logic'
export const handler = async (event: any = {}): Promise<any> => {
console.log(`Addition:${logic.add(2, 3)}`);
console.log(`SECURITY:${security.makesecure()}`);
};
src/layers/business-logic/logic.ts
export function add(a: number, b: number) {
return a + b;
}
src/layers/security-logic/logic.ts
export function makesecure() {
return 'SECURED';
}
lib/lambda-with-layers-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { NodejsFunction, NodejsFunctionProps } from 'aws-cdk-lib/aws-lambda-nodejs';
import path = require('path');
export class LambdaWithLayersStack extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const logicLayer = new lambda.LayerVersion(this, 'logic-layer', {
compatibleRuntimes: [
lambda.Runtime.NODEJS_18_X,
],
code: lambda.Code.fromAsset('src/layers/business-logic'),
description: 'Business Logic layer',
});
const securityLayer = new lambda.LayerVersion(this, 'security-layer', {
compatibleRuntimes: [
lambda.Runtime.NODEJS_18_X,
],
code: lambda.Code.fromAsset('src/layers/security-logic'),
description: 'Security Logic layer',
});
const nodeJsFnProps: NodejsFunctionProps = {
bundling: {
externalModules: [
'aws-sdk', // Use the 'aws-sdk' available in the Lambda runtime
],
},
runtime: lambda.Runtime.NODEJS_18_X,
timeout: cdk.Duration.minutes(3),
memorySize: 256,
};
const lambdaWithLayer = new NodejsFunction(this, 'lambdaWithLayer', {
entry: path.join(__dirname, '../src/lambdas', 'lambda.ts'),
...nodeJsFnProps,
functionName: 'lambdaWithLayer',
handler: 'handler',
layers: [ logicLayer, securityLayer ],
});
}
}
tsconfig.json
{
"compilerOptions": {
"target": "ES2020",
"module": "commonjs",
"lib": [
"es2020",
"dom"
],
"declaration": true,
"strict": true,
"noImplicitAny": true,
"strictNullChecks": true,
"noImplicitThis": true,
"alwaysStrict": true,
"noUnusedLocals": false,
"noUnusedParameters": false,
"noImplicitReturns": true,
"noFallthroughCasesInSwitch": false,
"inlineSourceMap": true,
"inlineSources": true,
"experimentalDecorators": true,
"strictPropertyInitialization": false,
"typeRoots": [
"./node_modules/@types"
],
"paths": {
"/opt/business-logic": ["./src/layers/business-logic/logic"],
"/opt/security-logic": ["./src/layers/security-logic/logic"]
}
},
"exclude": [
"node_modules",
"cdk.out"
]
}
lib/lambda-with-layers-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { NodejsFunction, NodejsFunctionProps } from 'aws-cdk-lib/aws-lambda-nodejs';
import path = require('path');
export class LambdaWithLayersStack extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const logicLayer = new lambda.LayerVersion(this, 'logic-layer', {
compatibleRuntimes: [
lambda.Runtime.NODEJS_18_X,
],
code: lambda.Code.fromAsset('src/layers/business-logic'),
description: 'Business Logic layer',
});
const securityLayer = new lambda.LayerVersion(this, 'security-layer', {
compatibleRuntimes: [
lambda.Runtime.NODEJS_18_X,
],
code: lambda.Code.fromAsset('src/layers/security-logic'),
description: 'Security Logic layer',
});
const nodeJsFnProps: NodejsFunctionProps = {
bundling: {
externalModules: [
'aws-sdk', // Use the 'aws-sdk' available in the Lambda runtime
],
},
runtime: lambda.Runtime.NODEJS_18_X,
timeout: cdk.Duration.minutes(3),
memorySize: 256,
};
const lambdaWithLayer = new NodejsFunction(this, 'lambdaWithLayer', {
entry: path.join(__dirname, '../src/lambdas', 'lambda.ts'),
...nodeJsFnProps,
functionName: 'lambdaWithLayer',
handler: 'handler',
layers: [ logicLayer, securityLayer ],
});
}
}
bin/cdktest.ts
#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { LambdaWithLayersStack } from '../lib/lambda-with-layers-stack';
const app = new cdk.App();
new LambdaWithLayersStack(app, 'LambdaWithLayersStack');
Deployed the function.
Ran cdk diff
. No differences were reported:
[+] Building 0.3s (14/14) FINISHED
docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.36kB 0.0s
=> [internal] load metadata for public.ecr.aws/sam/build-nodejs18.x:latest 0.2s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [ 1/10] FROM public.ecr.aws/sam/build-nodejs18.x:latest@sha256:fe81a3a950ea4d43320ee11ff228ad957679da2dcd2c7904e142c76bcdbc809a 0.0s
=> CACHED [ 2/10] RUN npm install --global yarn@1.22.5 0.0s
=> CACHED [ 3/10] RUN npm install --global pnpm@7.30.5 0.0s
=> CACHED [ 4/10] RUN npm install --global typescript 0.0s
=> CACHED [ 5/10] RUN npm install --global --unsafe-perm=true esbuild@0.21 0.0s
=> CACHED [ 6/10] RUN mkdir /tmp/npm-cache && chmod -R 777 /tmp/npm-cache && npm config --global set cache /tmp/npm-cache 0.0s
=> CACHED [ 7/10] RUN mkdir /tmp/yarn-cache && chmod -R 777 /tmp/yarn-cache && yarn config set cache-folder /tmp/yarn-cache 0.0s
=> CACHED [ 8/10] RUN mkdir /tmp/pnpm-cache && chmod -R 777 /tmp/pnpm-cache && pnpm config --global set store-dir /tmp/pnpm-cache 0.0s
=> CACHED [ 9/10] RUN npm config --global set update-notifier false 0.0s
=> CACHED [10/10] RUN /sbin/useradd -u 1000 user && chmod 711 / 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:a0093770fca3d866173e33525732120328cc5b175bbc5c59fe621bfa2bdeeb55 0.0s
=> => naming to docker.io/library/cdk-5944e75787ddbcab05dbee5d87a7fb3f8dccce03507b4909726365f73449fd9a 0.0s
View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/pxicut3a628vu2k50kgva7920
What's next:
View a summary of image vulnerabilities and recommendations → docker scout quickview
Bundling asset LambdaWithLayersStack/lambdaWithLayer/Code/Stage...
esbuild cannot run locally. Switching to Docker bundling.
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
asset-output/index.js 1.4kb
⚡ Done in 77ms
[Warning at /LambdaWithLayersStack] If you are relying on AWS SDK v2 to be present in the Lambda environment already, please explicitly configure a NodeJS runtime of Node 16 or lower. [ack: @aws-cdk/aws-lambda-nodejs:sdkV2NotInRuntime]
start: Building 7d53aa6c1c3b23e407fbcd6f9845f7f69da8535f3205739f7bfb5269b32b6635:current_account-current_region
success: Built 7d53aa6c1c3b23e407fbcd6f9845f7f69da8535f3205739f7bfb5269b32b6635:current_account-current_region
start: Publishing 7d53aa6c1c3b23e407fbcd6f9845f7f69da8535f3205739f7bfb5269b32b6635:current_account-current_region
success: Published 7d53aa6c1c3b23e407fbcd6f9845f7f69da8535f3205739f7bfb5269b32b6635:current_account-current_region
Hold on while we create a read-only change set to get a diff with accurate replacement information (use --no-change-set to use a less accurate but faster template-only diff)
Stack LambdaWithLayersStack
There were no differences
✨ Number of stacks with differences: 0
Tried deploying the stack again without changing the layer order in the CDK code. No deployment was done since there were no differences.
Ran cdk diff
again. No differences were reported.
Unsure if issue at your end is related to @aws-cdk/aws-lambda:recognizeLayerVersion
to false
in cdk.json. This is by default true
which executes the logic here to sort layers. I tried setting it to false
, but didn't observe differences when running cdk diff
.
Thanks, Ashish
Hi Ashish,
Thank you for looking into the issue. I understand that it might be difficult to reproduce the problem consistently, as the reordering of Lambda layers seems to occur intermittently. Here’s some more context based on our observations:
The problem is that the order of Lambda layers in the CloudFormation template for certain functions sometimes changes unexpectedly during deployment. The issue does not consistently affect the same Lambda function and may vary between different functions in each deployment. For example, during some deployments, one Lambda function might see its layers reordered, while in others, a different Lambda function shows the same behavior.
for example, here are two consecutive cdk diff
results:
Resources
[~] AWS::Lambda::Function _dev_editor_delete_actors deveditordeleteactorsF88F5F62
└─ [~] Layers
└─ @@ -1,4 +1,4 @@
[ ] [
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
[ ] ]
[~] AWS::Lambda::Function _dev_editor_delete_books deveditordeletebooks64D2EAD8
└─ [~] Layers
└─ @@ -1,4 +1,4 @@
[ ] [
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
[ ] ]
[~] AWS::Lambda::Function _dev_editor_get_statistics deveditorgetstatisticsB70FF0FD
└─ [~] Layers
└─ @@ -1,4 +1,4 @@
[ ] [
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[ ] ]
[~] AWS::Lambda::Function _dev_editor_patch_books deveditorpatchbooksAA79E65B
└─ [~] Layers
└─ @@ -1,4 +1,4 @@
[ ] [
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
[ ] ]
[~] AWS::Lambda::Function _dev_editor_post_actions deveditorpostactions827197CA
└─ [~] Layers
└─ @@ -1,4 +1,4 @@
[ ] [
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
[ ] ]
[~] AWS::Lambda::Function _dev_editor_post_actors deveditorpostactors78F9DF10
└─ [~] Layers
└─ @@ -1,4 +1,4 @@
[ ] [
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21"
[ ] ]
[~] AWS::Lambda::Function _dev_editor_post_actor_voices deveditorpostactorvoices69975E65
└─ [~] Layers
└─ @@ -1,4 +1,4 @@
[ ] [
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[ ] ]
[~] AWS::Lambda::Function _dev_editor_post_books deveditorpostbooks8ACBBF04
└─ [~] Layers
└─ @@ -1,5 +1,5 @@
[ ] [
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[ ] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
[ ] ]
[~] AWS::Lambda::Function _dev_editor_post_organizations deveditorpostorganizations89CB117D
└─ [~] Layers
└─ @@ -1,5 +1,5 @@
[ ] [
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[ ] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
[ ] ]
then, when I re-issue the cdk diff
right away:
Resources
[~] AWS::Lambda::Function _dev_editor_delete_books deveditordeletebooks64D2EAD8
└─ [~] Layers
└─ @@ -1,4 +1,4 @@
[ ] [
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
[ ] ]
[~] AWS::Lambda::Function _dev_editor_patch_books deveditorpatchbooksAA79E65B
└─ [~] Layers
└─ @@ -1,4 +1,4 @@
[ ] [
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
[ ] ]
[~] AWS::Lambda::Function _dev_editor_post_actors deveditorpostactors78F9DF10
└─ [~] Layers
└─ @@ -1,4 +1,4 @@
[ ] [
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21"
[ ] ]
[~] AWS::Lambda::Function _dev_editor_post_books deveditorpostbooks8ACBBF04
└─ [~] Layers
└─ @@ -1,5 +1,5 @@
[ ] [
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
[ ] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21"
[ ] ]
[~] AWS::Lambda::Function _dev_editor_post_organizations deveditorpostorganizations89CB117D
└─ [~] Layers
└─ @@ -1,5 +1,5 @@
[ ] [
[+] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[ ] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21",
[-] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
[ ] "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
[ ] ]
The layer ordering issue affects other lambdas.
Thanks for your support!
@eboily Thanks for your response. Could you please check if you have feature flag @aws-cdk/aws-lambda:recognizeLayerVersion
in cdk.json?
true
?false
, could you try setting it back to default value of true
?This feature flag affects the logic here to sort layers.
Thanks, Ashish
Hi Ashish,
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
was there and set to true.
Thanks,
Hi Ashish,
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
was there and set to true.Thanks,
- Edouard
@eboily Since the feature flag is set to true in your environment, it should have mitigated the issue. I'm unsure if it's feasible to reproduce the issue. Do you see any differences in your CDK project compared to if you create a new CDK project all together? Or, is it possible to share self-contained minimal CDK project alog with instructions, to try reproduce the issue?
Thanks, Ashish
This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.
Hi Ashish,
sorry for the answer’s delay, I’m fighting COVID. I’ll try to investigate more this instability and I’ll let you know.
Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.
Describe the bug
When deploying an AWS Lambda function with multiple layers using AWS CDK (v2), the order of layers in the CloudFormation template (AWS::Lambda::Function resource) changes between deployments. This causes the
cdk diff
command to show unnecessary changes, even though the functionality and deployment outcome remain the same.Regression Issue
Last Known Working CDK Version
No response
Expected Behavior
The order of layers in the AWS::Lambda::Function resource should remain consistent across deployments unless explicitly modified by the developer. This would prevent unnecessary changes from being reported in
cdk diff
.Current Behavior
The order of layers in the AWS::Lambda::Function resource changes arbitrarily between deployments. This results in diffs like the following:
Despite no functional changes, the difference in layer order causes unnecessary diffs.
Reproduction Steps
Possible Solution
Ensure that the order of Lambda layers is preserved and consistent between deployments unless the developer explicitly changes the order in the CDK code. The
cdk diff
command should only report changes when actual modifications (such as version changes or new layers) occur, and not simply due to arbitrary reordering of layers.Additional Information/Context
Environment:
Workaround:
Manually enforce layer order in the CDK code by specifying the order explicitly in the array when defining the Lambda function’s layers. However, this is cumbersome and not ideal for long-term maintenance.
Impact:
This issue leads to confusion and unnecessary diffs during deployment, which may slow down CI/CD pipelines and cause developers to spend time analyzing non-functional changes.
CDK CLI Version
2.161.1 (build 0a606c9)
Framework Version
No response
Node.js Version
v20.6.1
OS
macOS 15.0
Language
TypeScript
Language Version
Actually javascript, not typescript
Other information
No response