search

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.63k stars 3.91k forks source link

(aws-dynamodb): No replica created when new table created via Update event #12332

Closed tom-noble closed 3 years ago

tom-noble commented 3 years ago

Reproduction Steps

  1. Create a DynamoDB Table using AWS CDK with replication_regions set 
    aws_dynamodb.Table(
self,
"lcrCalculatorStatusTable",
partition_key={"name": "RunId", "type": aws_dynamodb.AttributeType.STRING},
sort_key=None,
billing_mode=aws_dynamodb.BillingMode.PAY_PER_REQUEST,
server_side_encryption=True,
replication_regions=["eu-central-1"],
)
  2. Synth and Deploy
  3. Update the partition key of the Table using AWS CDK (RunId to RUN_ID in this case) 
    aws_dynamodb.Table(
self,
"lcrCalculatorStatusTable",
partition_key={"name": "RUN_ID", "type": aws_dynamodb.AttributeType.STRING},
sort_key=None,
billing_mode=aws_dynamodb.BillingMode.PAY_PER_REQUEST,
server_side_encryption=True,
replication_regions=["eu-central-1"],
)
  4. Synth and Deploy (CDK attempts to update the table, and subsequently creates a new one)
5/28 | 12:29:06 PM | UPDATE_IN_PROGRESS   | AWS::DynamoDB::Table                | lcrCalculatorStatusTable (lcrCalculatorStatusTable99128E5D) Requested update requires the creation of a new physical resource; hence creating one.
5/28 | 12:29:06 PM | UPDATE_IN_PROGRESS   | AWS::DynamoDB::Table                | lcrCalculatorStatusTable (lcrCalculatorStatusTable99128E5D) Resource creation Initiated
7/28 | 12:29:37 PM | UPDATE_COMPLETE      | AWS::DynamoDB::Table                | lcrCalculatorStatusTable (lcrCalculatorStatusTable99128E5D)

What did you expect to happen?

Since a new table is created as part of the update, I expected the new table to be automatically replicated and the deployment to succeed.

What actually happened?

CDK attempts to create a new table with the new partition key but a replica is not created. This causes the DynamoDBReplica custom resource to hang in the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state.

Screenshot 2021-01-04 at 11 17 19

This eventually fails the deployment due to timeout

 15/28 | 1:00:51 PM | UPDATE_FAILED        | Custom::DynamoDBReplica             | lcrCalculatorStatusTable/Replicaeu-central-1/Default (lcrCalculatorStatusTableReplicaeucentral169ED77F2) Failed to update resource. Operation timed out
    new CustomResource (/tmp/jsii-kernel-ixys5V/node_modules/@aws-cdk/core/lib/custom-resource.js:28:25)
    \_ Table.createReplicaTables (/tmp/jsii-kernel-ixys5V/node_modules/@aws-cdk/aws-dynamodb/lib/table.js:875:35)
    \_ new Table (/tmp/jsii-kernel-ixys5V/node_modules/@aws-cdk/aws-dynamodb/lib/table.js:487:18)
    \_ /usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:3556:49
    \_ Kernel._wrapSandboxCode (/usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:4040:16)
    \_ Kernel._create (/usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:3556:26)
    \_ Kernel.create (/usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:3300:21)
    \_ KernelHost.processRequest (/usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13119:28)
    \_ KernelHost.run (/usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13057:14)
    \_ Immediate._onImmediate (/usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13060:37)

Environment

Other

I think the reason this happens is that despite creating a new table the event is marked as an Update (shown below).

Screenshot 2021-01-04 at 12 01 01

The code generated by CDK to handle DynamoDB replication only checks for Create and Delete events, so a replica isn't created (shown below).

Screenshot 2021-01-04 at 16 27 25

This is reinforced by the fact that deleting and rebuilding the stack fixes the issue. CDK attempts to Create the table from scratch, rather than Update an existing one, and a replica is created successfully. If I update the partition key of the table again, I expect to run into this issue again.

This is :bug: Bug Report

tom-noble commented 3 years ago

Any updates on this?

github-actions[bot] commented 3 years ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.