core: CrossRegionExportReader: Response object is too long

[antoncohen]

Describe the bug

When using crossRegionReferences: true the cdk deploy fails in the reader region with Response object is too long.

Similar to #23958, but on the reader side, likely with a different root cause.

Expected Behavior

I expect crossRegionReferences: true to work, and it be possible to reference variables across stacks in different regions.

Current Behavior

When referencing a variable in a stack in a different region, it fails in the reader region with this error:

testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 |  3/14 | 12:28:01 PM | CREATE_COMPLETE      | AWS::Lambda::Function           | Custom::CrossRegionExportReaderCustomResourceProvider/Handler (CustomCrossRegionExportReaderCustomResourceProviderHandler46647B68)
[12:28:03] Stack testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 |  3/14 | 12:28:06 PM | CREATE_IN_PROGRESS   | Custom::CrossRegionExportReader | ExportsReader/Resource/Default (ExportsReader8B249524)
[12:28:08] Stack testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
[12:28:13] Stack testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 |  3/14 | 12:28:15 PM | CREATE_FAILED        | Custom::CrossRegionExportReader | ExportsReader/Resource/Default (ExportsReader8B249524) Response object is too long.
    new CustomResource (/path/to/code/node_modules/aws-cdk-lib/core/lib/custom-resource.ts:135:21)
    \_ new ExportReader (/path/to/code/node_modules/aws-cdk-lib/core/lib/custom-resource-provider/cross-region-export-providers/export-reader-provider.ts:62:27)
    \_ Function.getOrCreate (/path/to/code/node_modules/aws-cdk-lib/core/lib/custom-resource-provider/cross-region-export-providers/export-reader-provider.ts:29:9)
    \_ ExportWriter.addToExportReader (/path/to/code/node_modules/aws-cdk-lib/core/lib/custom-resource-provider/cross-region-export-providers/export-writer-provider.ts:122:39)
    \_ ExportWriter.exportValue (/path/to/code/node_modules/aws-cdk-lib/core/lib/custom-resource-provider/cross-region-export-providers/export-writer-provider.ts:114:17)
    \_ createCrossRegionImportValue (/path/to/code/node_modules/aws-cdk-lib/core/lib/private/refs.ts:240:33)
    \_ resolveValue (/path/to/code/node_modules/aws-cdk-lib/core/lib/private/refs.ts:121:12)
    \_ resolveValue (/path/to/code/node_modules/aws-cdk-lib/core/lib/private/refs.ts:105:12)
    \_ resolveReferences (/path/to/code/node_modules/aws-cdk-lib/core/lib/private/refs.ts:34:24)
    \_ prepareApp (/path/to/code/node_modules/aws-cdk-lib/core/lib/private/prepare-app.ts:30:20)
    \_ synthesize (/path/to/code/node_modules/aws-cdk-lib/core/lib/private/synthesis.ts:45:13)
    \_ App.synth (/path/to/code/node_modules/aws-cdk-lib/core/lib/stage.ts:217:33)
    \_ process.<anonymous> (/path/to/code/node_modules/aws-cdk-lib/core/lib/app.ts:195:45)
    \_ Object.onceWrapper (node:events:628:26)
    \_ process.emit (node:events:513:28)
    \_ process.emit (node:domain:489:12)
    \_ process.emit.sharedData.processEmitHook.installedValue [as emit] (/path/to/code/node_modules/@cspotcode/source-map-support/source-map-support.js:745:40)
testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 |  3/14 | 12:28:16 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack      | testing-cdk-cross-region-references-as-example-for-github-issue-rootstack-us-west-2 The following resource(s) failed to create: [ExportsReader8B249524]. Rollback requested by user.

This is using Nested Stacks, and long stack names, with multiple cross-region references.

Reproduction Steps

Node.js: 18.15.0 CDK bin and lib: 2.73.0 TypeScript: 5.0.4 Command: npx cdk deploy --all --require-approval=never --verbose --debug --app 'npx ts-node app.ts'

app.ts:

#!/usr/bin/env npx ts-node

import * as awscdk from 'aws-cdk-lib'

const ACCOUNT_ID = '111222333444'
const STACK_PREFIX = 'testing-cdk-cross-region-references-as-example-for-github-issue'
// 8 or less works, 9 or more fails with "Response object is too long"
const NUMBER_OF_KEYS_TO_CREATE = 9

function createRootStack (app: awscdk.App, region: string): awscdk.Stack {
  const stackName = `${STACK_PREFIX}-rootstack-${region}`

  const props = {
    env: {
      account: ACCOUNT_ID,
      region: region
    },
    crossRegionReferences: true,
    synthesizer: new awscdk.LegacyStackSynthesizer()
  }

  const stack = new awscdk.Stack(app, stackName, props)

  return stack
}

function createPrimaryKeyStack (rootStack: awscdk.Stack, keyName: string): string {
  const nestedStack = new awscdk.NestedStack(rootStack, `${STACK_PREFIX}-nestedstack-${keyName}`)

  const keyProps = getKeyProps(keyName)
  const kmsCfnKey = new awscdk.aws_kms.CfnKey(nestedStack, `${STACK_PREFIX}-key-${keyName}`, keyProps)

  const aliasName = `alias/${STACK_PREFIX}-${keyName}`
  const aliasProps = getAliasProps(aliasName, kmsCfnKey.attrKeyId)
  new awscdk.aws_kms.CfnAlias(nestedStack, `${STACK_PREFIX}-alias-${keyName}`, aliasProps)

  return kmsCfnKey.attrArn
}

function createReplicaKeyStack (rootStack: awscdk.Stack, keyName: string, primaryKeyArn: string) {
  const nestedStack = new awscdk.NestedStack(rootStack, `${STACK_PREFIX}-nestedstack-${keyName}`)

  const keyProps = getReplicaKeyProps(keyName, primaryKeyArn)
  const kmsCfnKey = new awscdk.aws_kms.CfnReplicaKey(nestedStack, `${STACK_PREFIX}-key-${keyName}`, keyProps)

  const aliasName = `alias/${STACK_PREFIX}-${keyName}`
  const aliasProps = getAliasProps(aliasName, kmsCfnKey.attrKeyId)
  new awscdk.aws_kms.CfnAlias(nestedStack, `${STACK_PREFIX}-alias-${keyName}`, aliasProps)
}

function getKeyPolicy (): awscdk.aws_iam.PolicyDocument {
  const thisPrincipal = new awscdk.aws_iam.AccountPrincipal(ACCOUNT_ID)

  const policyStatement = new awscdk.aws_iam.PolicyStatement({
    sid: 'Allow all key access',
    effect: awscdk.aws_iam.Effect.ALLOW,
    actions: [
      'kms:*'
    ],
    resources: ['*'],
    principals: [thisPrincipal]
  })

  const policyDocument = new awscdk.aws_iam.PolicyDocument({ statements: [policyStatement] })

  return policyDocument
}

function getKeyProps (keyName: string): awscdk.aws_kms.CfnKeyProps {
  const policyDocument = getKeyPolicy()

  const keyProps: awscdk.aws_kms.CfnKeyProps = {
    description: `Testing cross-region references in CDK - ${keyName}`,
    keyPolicy: policyDocument,
    multiRegion: true,
    enableKeyRotation: false,
    enabled: true,
    keyUsage: 'ENCRYPT_DECRYPT',
    pendingWindowInDays: 7
  }

  return keyProps
}

function getReplicaKeyProps (keyName: string, primaryKeyArn: string): awscdk.aws_kms.CfnReplicaKeyProps {
  const policyDocument = getKeyPolicy()

  const keyProps: awscdk.aws_kms.CfnReplicaKeyProps = {
    description: `Testing cross-region references in CDK - ${keyName}`,
    primaryKeyArn: primaryKeyArn,
    keyPolicy: policyDocument,
    enabled: true,
    pendingWindowInDays: 7
  }

  return keyProps
}

function getAliasProps (aliasName:string, targetKeyId: string): awscdk.aws_kms.CfnAliasProps {
  const aliasProps: awscdk.aws_kms.CfnAliasProps = {
    aliasName: aliasName,
    targetKeyId: targetKeyId
  }

  return aliasProps
}

function main (args: Array<string> | undefined = undefined) {
  const app = new awscdk.App()

  const rootStackEast = createRootStack(app, 'us-east-1')
  const rootStackWest = createRootStack(app, 'us-west-2')

  for (let i = 1; i <= NUMBER_OF_KEYS_TO_CREATE; i++) {
    const primaryKeyArn = createPrimaryKeyStack(rootStackEast, i.toString())
    createReplicaKeyStack(rootStackWest, i.toString(), primaryKeyArn)
  }
}

if (require.main === module) {
  main()
}

At the top of app.ts the constant ACCOUNT_ID will need to be changed. The constant NUMBER_OF_KEYS_TO_CREATE can be lowered below 9 to not trigger the error.

I don't see any errors in the Lambda logs. Parameter Store values are written. And in the reader region the Parameter Store tags are set and removed, according to the Lambda logs.

Possible Solution

My guess is that it is triggered long stack names, including nested stacks creating even longer reference names.

Additional Information/Context

No response

CDK CLI Version

2.73.0 (build 43e681e)

Framework Version

No response

Node.js Version

18.15.0

OS

macOS

Language

Typescript

Language Version

TypeScript (5.0.4)

Other information

No response

[khushail]

Hi @antoncohen , stack names in AWS CDK have a maximum length of 128 characters. It might be better to have a look at name length to rule it out.

[antoncohen]

Hi @antoncohen , stack names in AWS CDK have a maximum length of 128 characters. It might be better to have a look at name length to rule it out.

It isn't a general stack name length issue, as far as I can tell. You can see the stack name in the error message and reproduction example code. The longest stack name is 83 characters.

Everything is created fine in the primary region (cross-region writer), it only fails in the replica region (cross-region reader). And if the NUMBER_OF_KEYS_TO_CREATE constant is changed to 8 or less it works. The stacks work. The cross-region references are what is failing when NUMBER_OF_KEYS_TO_CREATE is 9 or more.

[peterwoodworth]

We likely need a similar fix like this one that was abandoned. The issue is likely that the response in the custom resource handler is too long (over 4096 bytes)

[renschler]

running into this bug as well, what @peterwoodworth mentioned sounds right, when I look at the custom resource handler cloudwatch logs I don't see any errors, but the response object looks like it is likely over 4096 bytes. maybe we can store the response object in an s3 bucket and then just return a reference to the object in s3.

[renschler]

This behavior prevents me from being able to deploy to more regions and thus prevents me from spending more money on aws ;)

[colifran]

This is a CloudFormation limitation stemming from custom resource provider responses being limited to 4096 bytes - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/crpg-ref-responses.html. In the specific use case you've provided you can fix this by shortening the overall stack name by either shortening the stack prefix or by not using nested stacks which is adding to the stack name length. Longer term I'm not sure there is a good alternative here. We can't truncate or provide something like outputPaths since all of the imports are needed in the export reader response back to CloudFormation since the import of the exported values takes place using a getAtt: https://github.com/aws/aws-cdk/blob/2c53cf959d4de8a2d7cadfb24985087df1199305/packages/aws-cdk-lib/core/lib/custom-resource-provider/cross-region-export-providers/export-reader-provider.ts#L86-L89

[colifran]

Added documentation for this here and opened a new issue to track this for crossRegionReferences as a whole - https://github.com/aws/aws-cdk/issues/30119

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.