aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.64k stars 3.91k forks source link

CDk diff: Inconsistent Lambda Layer Ordering in CDK Diff #31703

Closed eboily closed 2 weeks ago

eboily commented 3 weeks ago

Describe the bug

When deploying an AWS Lambda function with multiple layers using AWS CDK (v2), the order of layers in the CloudFormation template (AWS::Lambda::Function resource) changes between deployments. This causes the cdk diff command to show unnecessary changes, even though the functionality and deployment outcome remain the same.

Regression Issue

Last Known Working CDK Version

No response

Expected Behavior

The order of layers in the AWS::Lambda::Function resource should remain consistent across deployments unless explicitly modified by the developer. This would prevent unnecessary changes from being reported in cdk diff.

Current Behavior

The order of layers in the AWS::Lambda::Function resource changes arbitrarily between deployments. This results in diffs like the following:

[~] AWS::Lambda::Function dev_editor_post_actors deveditorpostactors78F9DF10 
 └─ [~] Layers
     └─ @@ -1,4 +1,4 @@
        [ ] [
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxxx:layer:devawsHelpers:40",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxxx:layer:devcontract:21"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxxx:layer:devcontract:21",
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxxx:layer:devawsHelpers:40"
        [ ] ]

Despite no functional changes, the difference in layer order causes unnecessary diffs.

Reproduction Steps

1.  Create a Lambda function in CDK (v2) with multiple layers.
2.  Deploy the function.
3.  Run `cdk diff` and observe the ordering of the layers in the output.
4.  Deploy the stack again without changing the layer order in the CDK code.
5.  Run `cdk diff` again to see that the layer order has changed, triggering a false positive diff.

Possible Solution

Ensure that the order of Lambda layers is preserved and consistent between deployments unless the developer explicitly changes the order in the CDK code. The cdk diff command should only report changes when actual modifications (such as version changes or new layers) occur, and not simply due to arbitrary reordering of layers.

Additional Information/Context

Environment:

•   AWS CDK Version: 2.x
•   Language: JavaScript/TypeScript
•   Service: AWS Lambda, AWS SNS, CloudFormation

Workaround:

Manually enforce layer order in the CDK code by specifying the order explicitly in the array when defining the Lambda function’s layers. However, this is cumbersome and not ideal for long-term maintenance.

Impact:

This issue leads to confusion and unnecessary diffs during deployment, which may slow down CI/CD pipelines and cause developers to spend time analyzing non-functional changes.

CDK CLI Version

2.161.1 (build 0a606c9)

Framework Version

No response

Node.js Version

v20.6.1

OS

macOS 15.0

Language

TypeScript

Language Version

Actually javascript, not typescript

Other information

No response

ashishdhingra commented 2 weeks ago

@eboily Good afternoon. Thanks for opening the issue. Somehow, I'm unable to reproduce the issue. Below is CDK stack used with associated Lambda function and layers. Used CDK version 2.161.1 (build 0a606c9).

src/lambdas/lambda.ts

import * as logic from '/opt/business-logic';
import * as security from '/opt/security-logic'

export const handler = async (event: any = {}): Promise<any> => {
  console.log(`Addition:${logic.add(2, 3)}`);
  console.log(`SECURITY:${security.makesecure()}`);
};

src/layers/business-logic/logic.ts

export function add(a: number, b: number) {
  return a + b;
}

src/layers/security-logic/logic.ts

export function makesecure() {
  return 'SECURED';
}

lib/lambda-with-layers-stack.ts

import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { NodejsFunction, NodejsFunctionProps } from 'aws-cdk-lib/aws-lambda-nodejs';
import path = require('path');

export class LambdaWithLayersStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const logicLayer = new lambda.LayerVersion(this, 'logic-layer', {
      compatibleRuntimes: [
        lambda.Runtime.NODEJS_18_X,
      ],
      code: lambda.Code.fromAsset('src/layers/business-logic'),
      description: 'Business Logic layer',
    });

    const securityLayer = new lambda.LayerVersion(this, 'security-layer', {
      compatibleRuntimes: [
        lambda.Runtime.NODEJS_18_X,
      ],
      code: lambda.Code.fromAsset('src/layers/security-logic'),
      description: 'Security Logic layer',
    });

    const nodeJsFnProps: NodejsFunctionProps = {
      bundling: {
        externalModules: [
          'aws-sdk', // Use the 'aws-sdk' available in the Lambda runtime
        ],
      },
      runtime: lambda.Runtime.NODEJS_18_X,
      timeout: cdk.Duration.minutes(3),
      memorySize: 256,
    };

    const lambdaWithLayer = new NodejsFunction(this, 'lambdaWithLayer', {
      entry: path.join(__dirname, '../src/lambdas', 'lambda.ts'),
      ...nodeJsFnProps,
      functionName: 'lambdaWithLayer',
      handler: 'handler',
      layers: [ logicLayer, securityLayer ],
    });
  }
}

tsconfig.json

{
  "compilerOptions": {
    "target": "ES2020",
    "module": "commonjs",
    "lib": [
      "es2020",
      "dom"
    ],
    "declaration": true,
    "strict": true,
    "noImplicitAny": true,
    "strictNullChecks": true,
    "noImplicitThis": true,
    "alwaysStrict": true,
    "noUnusedLocals": false,
    "noUnusedParameters": false,
    "noImplicitReturns": true,
    "noFallthroughCasesInSwitch": false,
    "inlineSourceMap": true,
    "inlineSources": true,
    "experimentalDecorators": true,
    "strictPropertyInitialization": false,
    "typeRoots": [
      "./node_modules/@types"
    ],
    "paths": {
      "/opt/business-logic": ["./src/layers/business-logic/logic"],
      "/opt/security-logic": ["./src/layers/security-logic/logic"]
    }
  },
  "exclude": [
    "node_modules",
    "cdk.out"
  ]
}

lib/lambda-with-layers-stack.ts

import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { NodejsFunction, NodejsFunctionProps } from 'aws-cdk-lib/aws-lambda-nodejs';
import path = require('path');

export class LambdaWithLayersStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const logicLayer = new lambda.LayerVersion(this, 'logic-layer', {
      compatibleRuntimes: [
        lambda.Runtime.NODEJS_18_X,
      ],
      code: lambda.Code.fromAsset('src/layers/business-logic'),
      description: 'Business Logic layer',
    });

    const securityLayer = new lambda.LayerVersion(this, 'security-layer', {
      compatibleRuntimes: [
        lambda.Runtime.NODEJS_18_X,
      ],
      code: lambda.Code.fromAsset('src/layers/security-logic'),
      description: 'Security Logic layer',
    });

    const nodeJsFnProps: NodejsFunctionProps = {
      bundling: {
        externalModules: [
          'aws-sdk', // Use the 'aws-sdk' available in the Lambda runtime
        ],
      },
      runtime: lambda.Runtime.NODEJS_18_X,
      timeout: cdk.Duration.minutes(3),
      memorySize: 256,
    };

    const lambdaWithLayer = new NodejsFunction(this, 'lambdaWithLayer', {
      entry: path.join(__dirname, '../src/lambdas', 'lambda.ts'),
      ...nodeJsFnProps,
      functionName: 'lambdaWithLayer',
      handler: 'handler',
      layers: [ logicLayer, securityLayer ],
    });
  }
}

bin/cdktest.ts

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { LambdaWithLayersStack } from '../lib/lambda-with-layers-stack';

const app = new cdk.App();

new LambdaWithLayersStack(app, 'LambdaWithLayersStack');
  1. Deployed the function.

  2. Ran cdk diff. No differences were reported:

    [+] Building 0.3s (14/14) FINISHED                                                                                                                                                             
    docker:desktop-linux
     => [internal] load build definition from Dockerfile                                                                                                                                                           0.0s
     => => transferring dockerfile: 1.36kB                                                                                                                                                                         0.0s
     => [internal] load metadata for public.ecr.aws/sam/build-nodejs18.x:latest                                                                                                                                    0.2s
     => [internal] load .dockerignore                                                                                                                                                                              0.0s
     => => transferring context: 2B                                                                                                                                                                                0.0s
     => [ 1/10] FROM public.ecr.aws/sam/build-nodejs18.x:latest@sha256:fe81a3a950ea4d43320ee11ff228ad957679da2dcd2c7904e142c76bcdbc809a                                                                            0.0s
     => CACHED [ 2/10] RUN npm install --global yarn@1.22.5                                                                                                                                                        0.0s
     => CACHED [ 3/10] RUN npm install --global pnpm@7.30.5                                                                                                                                                        0.0s
     => CACHED [ 4/10] RUN npm install --global typescript                                                                                                                                                         0.0s
     => CACHED [ 5/10] RUN npm install --global --unsafe-perm=true esbuild@0.21                                                                                                                                    0.0s
     => CACHED [ 6/10] RUN mkdir /tmp/npm-cache &&     chmod -R 777 /tmp/npm-cache &&     npm config --global set cache /tmp/npm-cache                                                                             0.0s
     => CACHED [ 7/10] RUN mkdir /tmp/yarn-cache &&     chmod -R 777 /tmp/yarn-cache &&     yarn config set cache-folder /tmp/yarn-cache                                                                           0.0s
     => CACHED [ 8/10] RUN mkdir /tmp/pnpm-cache &&     chmod -R 777 /tmp/pnpm-cache &&     pnpm config --global set store-dir /tmp/pnpm-cache                                                                     0.0s
     => CACHED [ 9/10] RUN npm config --global set update-notifier false                                                                                                                                           0.0s
     => CACHED [10/10] RUN /sbin/useradd -u 1000 user && chmod 711 /                                                                                                                                               0.0s
     => exporting to image                                                                                                                                                                                         0.0s
     => => exporting layers                                                                                                                                                                                        0.0s
     => => writing image sha256:a0093770fca3d866173e33525732120328cc5b175bbc5c59fe621bfa2bdeeb55                                                                                                                   0.0s
     => => naming to docker.io/library/cdk-5944e75787ddbcab05dbee5d87a7fb3f8dccce03507b4909726365f73449fd9a                                                                                                        0.0s
    
    View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/pxicut3a628vu2k50kgva7920
    
    What's next:
        View a summary of image vulnerabilities and recommendations → docker scout quickview 
    Bundling asset LambdaWithLayersStack/lambdaWithLayer/Code/Stage...
    esbuild cannot run locally. Switching to Docker bundling.
    WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
    
      asset-output/index.js  1.4kb
    
    ⚡ Done in 77ms
    [Warning at /LambdaWithLayersStack] If you are relying on AWS SDK v2 to be present in the Lambda environment already, please explicitly configure a NodeJS runtime of Node 16 or lower. [ack: @aws-cdk/aws-lambda-nodejs:sdkV2NotInRuntime]
    start: Building 7d53aa6c1c3b23e407fbcd6f9845f7f69da8535f3205739f7bfb5269b32b6635:current_account-current_region
    success: Built 7d53aa6c1c3b23e407fbcd6f9845f7f69da8535f3205739f7bfb5269b32b6635:current_account-current_region
    start: Publishing 7d53aa6c1c3b23e407fbcd6f9845f7f69da8535f3205739f7bfb5269b32b6635:current_account-current_region
    success: Published 7d53aa6c1c3b23e407fbcd6f9845f7f69da8535f3205739f7bfb5269b32b6635:current_account-current_region
    Hold on while we create a read-only change set to get a diff with accurate replacement information (use --no-change-set to use a less accurate but faster template-only diff)
    Stack LambdaWithLayersStack
    There were no differences
    
    ✨  Number of stacks with differences: 0
  3. Tried deploying the stack again without changing the layer order in the CDK code. No deployment was done since there were no differences.

  4. Ran cdk diff again. No differences were reported.

Unsure if issue at your end is related to @aws-cdk/aws-lambda:recognizeLayerVersion to false in cdk.json. This is by default true which executes the logic here to sort layers. I tried setting it to false, but didn't observe differences when running cdk diff.

Thanks, Ashish

eboily commented 2 weeks ago

Hi Ashish,

Thank you for looking into the issue. I understand that it might be difficult to reproduce the problem consistently, as the reordering of Lambda layers seems to occur intermittently. Here’s some more context based on our observations:

The problem is that the order of Lambda layers in the CloudFormation template for certain functions sometimes changes unexpectedly during deployment. The issue does not consistently affect the same Lambda function and may vary between different functions in each deployment. For example, during some deployments, one Lambda function might see its layers reordered, while in others, a different Lambda function shows the same behavior.

for example, here are two consecutive cdk diff results:

Resources
[~] AWS::Lambda::Function _dev_editor_delete_actors deveditordeleteactorsF88F5F62
 └─ [~] Layers
     └─ @@ -1,4 +1,4 @@
        [ ] [
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
        [ ] ]
[~] AWS::Lambda::Function _dev_editor_delete_books deveditordeletebooks64D2EAD8
 └─ [~] Layers
     └─ @@ -1,4 +1,4 @@
        [ ] [
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
        [ ] ]
[~] AWS::Lambda::Function _dev_editor_get_statistics deveditorgetstatisticsB70FF0FD
 └─ [~] Layers
     └─ @@ -1,4 +1,4 @@
        [ ] [
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [ ] ]
[~] AWS::Lambda::Function _dev_editor_patch_books deveditorpatchbooksAA79E65B
 └─ [~] Layers
     └─ @@ -1,4 +1,4 @@
        [ ] [
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
        [ ] ]
[~] AWS::Lambda::Function _dev_editor_post_actions deveditorpostactions827197CA
 └─ [~] Layers
     └─ @@ -1,4 +1,4 @@
        [ ] [
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
        [ ] ]
[~] AWS::Lambda::Function _dev_editor_post_actors deveditorpostactors78F9DF10
 └─ [~] Layers
     └─ @@ -1,4 +1,4 @@
        [ ] [
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21"
        [ ] ]
[~] AWS::Lambda::Function _dev_editor_post_actor_voices deveditorpostactorvoices69975E65
 └─ [~] Layers
     └─ @@ -1,4 +1,4 @@
        [ ] [
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [ ] ]
[~] AWS::Lambda::Function _dev_editor_post_books deveditorpostbooks8ACBBF04
 └─ [~] Layers
     └─ @@ -1,5 +1,5 @@
        [ ] [
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [ ]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
        [ ] ]
[~] AWS::Lambda::Function _dev_editor_post_organizations deveditorpostorganizations89CB117D
 └─ [~] Layers
     └─ @@ -1,5 +1,5 @@
        [ ] [
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [ ]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
        [ ] ]

then, when I re-issue the cdk diff right away:

Resources
[~] AWS::Lambda::Function _dev_editor_delete_books deveditordeletebooks64D2EAD8
 └─ [~] Layers
     └─ @@ -1,4 +1,4 @@
        [ ] [
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
        [ ] ]
[~] AWS::Lambda::Function _dev_editor_patch_books deveditorpatchbooksAA79E65B
 └─ [~] Layers
     └─ @@ -1,4 +1,4 @@
        [ ] [
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30"
        [ ] ]
[~] AWS::Lambda::Function _dev_editor_post_actors deveditorpostactors78F9DF10
 └─ [~] Layers
     └─ @@ -1,4 +1,4 @@
        [ ] [
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21"
        [ ] ]
[~] AWS::Lambda::Function _dev_editor_post_books deveditorpostbooks8ACBBF04
 └─ [~] Layers
     └─ @@ -1,5 +1,5 @@
        [ ] [
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40",
        [ ]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21"
        [ ] ]
[~] AWS::Lambda::Function _dev_editor_post_organizations deveditorpostorganizations89CB117D
 └─ [~] Layers
     └─ @@ -1,5 +1,5 @@
        [ ] [
        [+]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [ ]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devcontract:21",
        [-]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:dev:30",
        [ ]   "arn:aws:lambda:us-east-1:xxxxxxxxxxxxx:layer:devawsHelpers:40"
        [ ] ]

The layer ordering issue affects other lambdas.

Thanks for your support!

ashishdhingra commented 2 weeks ago

@eboily Thanks for your response. Could you please check if you have feature flag @aws-cdk/aws-lambda:recognizeLayerVersion in cdk.json?

This feature flag affects the logic here to sort layers.

Thanks, Ashish

eboily commented 2 weeks ago

Hi Ashish,

"@aws-cdk/aws-lambda:recognizeLayerVersion": true, was there and set to true.

Thanks,

ashishdhingra commented 2 weeks ago

Hi Ashish,

"@aws-cdk/aws-lambda:recognizeLayerVersion": true, was there and set to true.

Thanks,

  • Edouard

@eboily Since the feature flag is set to true in your environment, it should have mitigated the issue. I'm unsure if it's feasible to reproduce the issue. Do you see any differences in your CDK project compared to if you create a new CDK project all together? Or, is it possible to share self-contained minimal CDK project alog with instructions, to try reproduce the issue?

Thanks, Ashish

github-actions[bot] commented 2 weeks ago

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

eboily commented 2 weeks ago

Hi Ashish,

sorry for the answer’s delay, I’m fighting COVID. I’ll try to investigate more this instability and I’ll let you know.

github-actions[bot] commented 2 weeks ago

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.