search

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.63k stars 3.91k forks source link

core: referencing arrays/lists when using `crossRegionReferences` fails with "SerializationException: Start of list found where not expected" #31655

Open nathanielks opened 3 weeks ago

nathanielks commented 3 weeks ago

Describe the bug

When creating stacks with crossRegionReferences enabled, and the stack exposes a list/array value that is to be read in another region, the Custom::CrossRegionExportWriter resource will fail during creation with the error "SerializationException: Start of list found where not expected". Reading string values works fine, but if it's an array/list, it will fail.

Regression Issue

Last Known Working CDK Version

No response

Expected Behavior

I expect to be able to reference lists/arrays from a stack in another region like I would strings.

Current Behavior

The Custom::CrossRegionExportWriter resource fails during creation with the error "SerializationException: Start of list found where not expected":

BugVPCStack-us-west-2
BugVPCStack-us-west-2: deploying... [1/4]
Bug-VPC1-Requests-VPC2:  start: Building bc980b3b9776ac14eeed5eeecde033423deae6a101589e745c39d5e53e119d87:current_account-us-west-2
Bug-VPC1-Requests-VPC2:  success: Built bc980b3b9776ac14eeed5eeecde033423deae6a101589e745c39d5e53e119d87:current_account-us-west-2
Bug-VPC2-Accepts-VPC1:  start: Building a7bb53c2fb4a92a7dcc01a0863f4145f4161926cdb3d34becc9fe4a65ce54566:current_account-us-east-1
Bug-VPC2-Accepts-VPC1:  success: Built a7bb53c2fb4a92a7dcc01a0863f4145f4161926cdb3d34becc9fe4a65ce54566:current_account-us-east-1
BugVPCStack-us-west-2: creating CloudFormation changeset...
3:14:45 PM | UPDATE_FAILED        | Custom::CrossRegionExportWriter       | ExportsWriteruseast10F67B507DDE2E818
Received response status [FAILED] from custom resource. Message returned: SerializationException: Start of list found where not expected
at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/index.js:7021:14)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
at async Promise.all (index 0)
at async u (/var/task/index.js:3:330) (RequestId: f443b2b6-c5f3-49e6-98ce-eec9f39ef19d)

 ❌  BugVPCStack-us-west-2 failed: Error: The stack named BugVPCStack-us-west-2 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: SerializationException: Start of list found where not expected
    at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
    at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
    at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/index.js:7021:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async Promise.all (index 0)
    at async u (/var/task/index.js:3:330) (RequestId: f443b2b6-c5f3-49e6-98ce-eec9f39ef19d)
    at FullCloudFormationDeployment.monitorDeployment (/Users/aang/Code/Work/repos/core/cdk/Vpc-Peering/node_modules/aws-cdk/lib/index.js:458:10567)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/Users/aang/Code/Work/repos/core/cdk/Vpc-Peering/node_modules/aws-cdk/lib/index.js:461:200334)
    at async /Users/aang/Code/Work/repos/core/cdk/Vpc-Peering/node_modules/aws-cdk/lib/index.js:461:181756

 ❌ Deployment failed: Error: The stack named BugVPCStack-us-west-2 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: SerializationException: Start of list found where not expected
    at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
    at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
    at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/index.js:7021:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async Promise.all (index 0)
    at async u (/var/task/index.js:3:330) (RequestId: f443b2b6-c5f3-49e6-98ce-eec9f39ef19d)
    at FullCloudFormationDeployment.monitorDeployment (/Users/aang/Code/Work/repos/core/cdk/Vpc-Peering/node_modules/aws-cdk/lib/index.js:458:10567)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/Users/aang/Code/Work/repos/core/cdk/Vpc-Peering/node_modules/aws-cdk/lib/index.js:461:200334)
    at async /Users/aang/Code/Work/repos/core/cdk/Vpc-Peering/node_modules/aws-cdk/lib/index.js:461:181756

The stack named BugVPCStack-us-west-2 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: SerializationException: Start of list found where not expected
    at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
    at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
    at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/index.js:7021:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async Promise.all (index 0)
    at async u (/var/task/index.js:3:330) (RequestId: f443b2b6-c5f3-49e6-98ce-eec9f39ef19d)

Reproduction Steps

By default, the app will be created in a way that does not trigger the bug. To deploy the app as-is:

cdk bootstrap
cdk deploy --all --app 'node app.ts'

This should create the stacks successfully. Then run another deployment with a context variable:

cdk deploy --all --app 'node app.ts' --context readIPv6=true

This will instruct the app to reference list values in a cross-region manager and trigger the bug.

The bug occurs whether you create the stack with or without the context variable set; that is to say, if you create the stack with the context variable, the bug will also be triggered in that circumstance.

app.ts:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';

interface VpcStackProps extends cdk.StackProps {
  readonly cidr: string;
}

class VpcStack extends cdk.Stack {
  public readonly vpcId: string;
  public readonly vpcCidrIpV4: string;
  public readonly vpcCidrIpV6: string;
  public readonly vpcCidrIpV6List: string[];

  public constructor(scope: cdk.App, id: string, props: VpcStackProps) {
    super(scope, id, props);

    const vpc = new ec2.Vpc(this, 'Bug VPC', {
      ipAddresses: ec2.IpAddresses.cidr(props.cidr),
      ipProtocol: ec2.IpProtocol.DUAL_STACK,
      createInternetGateway: false,
      maxAzs: 3,
      subnetConfiguration: [
        {
          cidrMask: 28,
          name: 'private',
          subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
        }
      ]
    })

    this.vpcId = vpc.vpcId
    this.vpcCidrIpV4 = vpc.vpcCidrBlock

    // Both the selected value and the original list value will trigger the bug
    this.vpcCidrIpV6 = cdk.Fn.select(0, vpc.vpcIpv6CidrBlocks)
    this.vpcCidrIpV6List = vpc.vpcIpv6CidrBlocks
  }
}

function createVpcStack(region: string, cidr: string) {
  const stack = new VpcStack(app, `BugVPCStack-${region}`, {
    env: { region },
    crossRegionReferences: true,
    cidr
  })

  return stack
}

const ec2 = cdk.aws_ec2
const app = new cdk.App();

const vpc1Stack = createVpcStack('us-west-2', '10.16.0.0/16')
const vpc2Stack = createVpcStack('us-east-1', '10.17.0.0/16')

const requesterStack = new cdk.Stack(app, `Bug-VPC1-Requests-VPC2`, {
  env: { region: vpc1Stack.region },
  crossRegionReferences: true
})

const readIPv6 = app.node.tryGetContext('readIPv6')
const values = [
  vpc1Stack.vpcId!,
  vpc1Stack.vpcCidrIpV4!,
  vpc2Stack.vpcId!,
  vpc2Stack.vpcCidrIpV4!,
  ...(readIPv6 
    ? [
      vpc1Stack.vpcCidrIpV6!,
      vpc2Stack.vpcCidrIpV6!,
    ]
    : [])
]

new cdk.aws_ssm.StringListParameter(requesterStack, 'sourceVpc', {
  stringListValue: values,
})

if (readIPv6) new cdk.aws_ssm.StringListParameter(requesterStack, 'sourceVpcIpv6', {
  stringListValue: vpc2Stack.vpcCidrIpV6List!,
})

const accepterStack = new cdk.Stack(app, `Bug-VPC2-Accepts-VPC1`, {
  env: { region: vpc2Stack.region },
  crossRegionReferences: true
})

new cdk.aws_ssm.StringListParameter(accepterStack, 'sourceVpc', {
  stringListValue: values,
})

if (readIPv6) new cdk.aws_ssm.StringListParameter(accepterStack, 'sourceVpcIpv6', {
  stringListValue: vpc1Stack.vpcCidrIpV6List!,
})

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.158.0

Framework Version

No response

Node.js Version

20.15.0

OS

macOS 14.6.1

Language

TypeScript

Language Version

5.5.4

Other information

No response

pahud commented 3 weeks ago

Reproducible. This seems a bug to me.

Let me simplify it with this PoC

VpcStack

export interface VpcStackProps extends cdk.StackProps {
  readonly cidr?: string;
}
export class VpcStack extends Stack {
  public readonly vpcId: string;
  public readonly vpcCidrIpV4: string;
  public readonly vpcCidrIpV6: string;
  public readonly vpcCidrIpV6List: string[];
  constructor(scope: Construct, id: string, props: VpcStackProps) {
    super(scope, id, props);

    const vpc = new ec2.Vpc(this, 'Bug VPC', {
      ipAddresses: ec2.IpAddresses.cidr('10.16.0.0/16'),
      ipProtocol: ec2.IpProtocol.DUAL_STACK,
      createInternetGateway: false,
      maxAzs: 3,
      subnetConfiguration: [
        {
          cidrMask: 28,
          name: 'private',
          subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
        }
      ]
    })

    this.vpcId = vpc.vpcId
    this.vpcCidrIpV4 = vpc.vpcCidrBlock

    this.vpcCidrIpV6 = cdk.Fn.select(0, vpc.vpcIpv6CidrBlocks);
    new CfnOutput(this, 'vpcCidrIpV6', { value: this.vpcCidrIpV6  })
  }
}

It's fine when just deploy the vpc stack

new VpcStack(app, 'vpc-stack', { env });
Outputs:
vpc-stack.vpcCidrIpV6 = 2600:1f18:601f:ec00::/56

Now if we define a consumer stack

export interface ConsumerStackProps extends StackProps {
  readonly vpcCidrIpV6: string;
}

export class ConsumerStack extends Stack {
  constructor(scope: Construct, id: string, props: ConsumerStackProps) {
    super(scope, id, props);

    new CfnOutput(this, 'MyListOutput', { value: props.vpcCidrIpV6 })
  }
}

And enable crossRegionReferences:

const stack = new VpcStack(app, 'vpc-stack', { env, crossRegionReferences: true });

new ConsumerStack(app, 'consumer-stack', { env: {
    account: process.env.CDK_DEFAULT_ACCOUNT,
    region: 'us-west-2',
} , vpcCidrIpV6: stack.vpcCidrIpV6, crossRegionReferences: true })

The writer CR would fail with SerializationException:

11:03:42 PM | CREATE_FAILED | Custom::CrossRegionExportWriter | ExportsWriteruswes...0/Resource /Default Received response status [FAILED] from custom resource. Message returned: SerializationException: Start of l ist found where not expected at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.j s:840:20) at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5 at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/index.js:7021:14) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20 at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18 at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38

Making this a p1.