AWS::DynamoDB::Table Global Tables

[PatMyron]

2. Scope

c) new attribute for existing resource is desired

5. Links

https://aws.amazon.com/dynamodb/global-tables/ https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GlobalTables.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dynamodb-table.html https://aws.amazon.com/blogs/aws/new-convert-your-single-region-amazon-dynamodb-tables-to-global-tables/

---

https://forums.aws.amazon.com/thread.jspa?threadID=288408 https://stackoverflow.com/questions/47984111/dynamodb-global-tables-using-cloudformation https://github.com/kspurrier/cfn-dynamodb-global-table

6. Category:

3. DB (RDS, DynamoDB...)

[kayengar]

Is there any traction on this issue? We have been using some lambda backed custom resources for the past 2 years and this would be really good to have.

[twratl]

With the new version of global tables this gets even easier I believe. I have custom resources that do the below. Would be great to see it baked into a native resource.

In the "primary" region deploy the dynamo db tables via CloudFormation. They will need a name provided since we have to have a deterministic name as all the regional replicas must use the same name. No global tables yet.

Then when you deploy your CloudFormation template to "secondary" regions instead of creating the dynamo tables you create a resource which effectively adds the region in question as a global table replica to the table in the primary region via an UpdateTable call on the table in the primary region. My custom resource handles this.

Upon updates and deletes, remove the secondary region as required.

[luiseduardocolon]

Folks, I just got an update on this today. Looks like it might be ready to be released sometime in the next two months. Stay tuned.

[jthomerson]

@luiseduardocolon is there any way to give customer feedback on the design of that resource? Or to get involved in testing it? I have dozens of DynamoDB tables (don't tell Rick Houlihan) that all use global tables. We've had two different iterations of custom resources for this that have each had their own issues.

I'd love to be able to give feedback early rather than see it shipped and it's still missing one fundamental piece that we need or something like that. Thanks for the update, and thanks for considering this.

[alex131290]

Is there any progress with it?

[luiseduardocolon]

Hey @alex131290 - this is progressing nicely... stay tuned.

[PatrykMilewski]

Why can't you guys just give us estimated date when it goes on production? Some of our stuff are blocked by that and I'm not sure if I should wait those 1-2 weeks or simply do workarounds.

[jthomerson]

Hey @luiseduardocolon you mentioned four weeks ago that this is progressing, but we don't have any idea if that means it might be ready in two weeks or 16 months. Can we at least get a t-shirt size estimate? My custom resources that make global tables still bake the old global tables, and I really want to use the new ones that don't have the AWS replication fields and the double updates ... but I don't know whether to invest in that work or not. If the official support will be live in the next 2 to 6 weeks, I probably just wait. If it's longer, I may need to go ahead and invest.

[nukepuppy]

Same here.. @luiseduardocolon could you provide an update? It seems as if it's going really well after all this time.. perhaps even a ballpark estimate so others know where to invest? the workaround for this is a lot of spaghetti that doesn't need to happen.

[ohadbenita]

Would be happy to know as well.

[luiseduardocolon]

Hello there @nukepuppy @ohadbenita @jthomerson @PatrykMilewski - first, my sincere apologies for the delay in updating things. Luckily for us work has continued on this while I've been working in other projects. Today, I am moving this to "Coming Soon". This is to reflect that we have deployed this to our production servers, but we don't have docs quite ready yet. We believe docs should be finished in the next 2 weeks, possibly as soon as 10 days from now.

[prasad-mhatre-incontact]

eagerly waiting for this feature, as this will make our life simpler. Please keep us posted.

[ohadbenita]

Waiting for this as well, please let us know as soon as this is out @luiseduardocolon

[luiseduardocolon]

This is in the final stages; we are very focused on delivering this ASAP. We have encountered late errors as we test edge cases. Also, we are planning to release this and allow the community to help us iterate and improve it quickly after release. Ping me if you want more details.

[prasad-mhatre-incontact]

Hi @luiseduardocolon , I would like have it look. can you please share details for integrating this and testing it.

[mm326]

Hi @luiseduardocolon just wanted to clarify something, is Cloudformation currently supporting Global tables but just the documentation isn't ready?

[seeebiii]

@luiseduardocolon I'd be interested in receiving more details as well :-) How do we get them?

[prasad-mhatre-incontact]

@luiseduardocolon : Can you please share details for testing?

[luiseduardocolon]

Trying to get an update yet this week here.

[onv]

@luiseduardocolon do you have any updates?

[elifitch]

@luiseduardocolon If you're looking for the community to help iterate and improve it, I'm sure there's at least one team at Mapbox that would jump at the chance for early access and would happily work with y'all to provide feedback.

[JM-TW]

i need this asap, in the meantime if i deploy and then configure the ddbb table as global, and re-deploy will it remove the global table settings if im not updating anything in the template for the table?

[mariaolivero-RL]

Wow... this has been 2 weeks out since April. Any updates?

[onv]

@luiseduardocolon is it so hard to give some information and updated ETA for so important feature?

[colinhuckstep]

I'm still waiting for this as well. I've had to write a custom resource for the time being.

[prasad-mhatre-incontact]

I'm still waiting for this as well. I've had to write a custom resource for the time being.

Same here

[idun-admin]

Hi, just bumping this thread to see if there's anything forthcoming. Judging by the history, I assume not?

[ajhool]

Excited for this to arrive. Upon arrival, it would be really helpful if there were instructions on how to "upgrade" an existing dynamodb table to leverage this global table feature -- or, if it isn't possible to do so, that would be good to know, too. Upgrading a regional table to global seems to be possible [1].

To restate: existing dynamodb table deployed via cloudformation -> enable global table on that dynamodb table using this new cloudformation feature -> keep data.

[1] New – Convert Your Single-Region Amazon DynamoDB Tables to Global Tables, https://aws.amazon.com/blogs/aws/new-convert-your-single-region-amazon-dynamodb-tables-to-global-tables/

[mkamioner]

Any updates @luiseduardocolon ?

[djfurman]

Plus 1 to this; I want to make sure that I can get accurate stream ARNs from the CFTs to import in other CFT where my lambda functions need to be processing in each region. Do we have line of sight?

[adriaan-pelzer]

I've been following this for quite some time, and I'm afraid AWS has gone all quiet on us (as they often do).

Here's how I solved it in the end:

Run CLI commands in my pipelines (after cfn deploy) to loop through all desired replicated regions, check if they already exist, create them if they don't, and likewise check if any deployed regions should be removed.

In lambda@edge, compare the AWS_REGION env var with the list of replicated regions, pick the one with the longest string overlap, and use that to build the DynamoDB ARN.

[glb]

@luiseduardocolon Will the CloudFormation support allow customers to create and manage 2019.11.21 tables or is it limited to 2017.11.29? I was quite surprised to discover that the SDK only has support for 2017 at this time.

[luiseduardocolon]

Okay, it took me a couple of days to dig for an answer here folks. We are actively working with the DDB team on this and, barring any unforeseen surprises, this looks like it will drop by the end of March. As for the tables version, it looks like they are targeting support for the v2, released on 2019. However, I don't know why v2 is not supported on the SDK yet; I assume that support will happen and the CloudFormation resource will use that support, since we use the SDK (as you would expect) to implement the native resources too. For the SDK specifics, you may want to open a ticket and/or check with either the DDB team or the SDK team directly. Let me know if you run into a wall :) And thanks for your patience. Any other questions? - Luis

[glb]

Thank you @luiseduardocolon ! I think I've figured out the confusion our team had with versioning -- still confirming but apparently CreateGlobalTable is the 2017 way and UpdateTable is the 2019 way.

[onv]

Hi @luiseduardocolon, maybe you already know the date when GlobalTables support will be added to CFN? Just 2 weeks left till the end of March.

[RobertoC27]

Any updates on ETA? My team is starting to look into and this and investing into custom resources when this is 'so close' seems counterintuitive.

[colinhuckstep]

Any updates on ETA? My team is starting to look into and this and investing into custom resources when this is 'so close' seems counterintuitive.

I gave up waiting and built a custom resource. Not sure what your strategy on that would be, but I built a generic custom resource for calling an Ansible playbook via the Ansible Tower API. We've ended up using it for creating a whole lot of Service Catalog products that allow our Dev teams to call Ansible Playbooks via CloudFormation. I created an Invoke Tower Lambda that the service catalog products use. The product template just gives the template ID and extra_vars to Ansible Tower API. It's allowed us to consume anything that can be done on the AWS CLI via CloudFormation. Really anything you can do with Ansible. I've got products for CRUD AD DNS records, trigging code build jobs.

I've been referring to Ansible but insert your Automation Engine of choice. All of this to say, If it's got a REST API available spend the time to make a custom resource. If you make it generic enough to call whatever you need from your Automation Engine then it's not going to be a waste of time. It's allowed us to make things available via service catalog products months, or in the case of DDB global tables, a year ahead of it being available in CloudFormation.

[RobertoC27]

Hey @colinhuckstep thanks for pointing that out. You're right, generic enough custom resources can work and we have another case where we could benefit from them (CloudHSM is still not supported by cfn), but one less thing to manage its always nice. Service catalog + custom resources seems like a nice combo for sure, we'll look into thanks!

We are a small shop but given that level of flexibility sure could benefit us further down the line.

[callum-p]

I needed to use this today and after reading through this issue I'm quite disappointed at the '2 more weeks' rhetoric 2 years later. I'll be creating a custom resource too today I guess :(

[endgame]

@callum-p Not sure if it's any use to you, but I noticed that AWS CDK bundles a custom resource to implement the global table option in its higher-level DynamoDB constructs.

[callum-p]

@callum-p Not sure if it's any use to you, but I noticed that AWS CDK bundles a custom resource to implement the global table option in its higher-level DynamoDB constructs.

Thanks @endgame - got what I needed via the CDK.

[avinayurfriend]

@callum-p Not sure if it's any use to you, but I noticed that AWS CDK bundles a custom resource to implement the global table option in its higher-level DynamoDB constructs.

Thanks @endgame - got what I needed via the CDK.

@callum-p can you share your cdk implementation snippet. Would help me a lot. Thanks :)

[callum-p]

@callum-p can you share your cdk implementation snippet. Would help me a lot. Thanks :)

@avinayurfriend

import * as dynamodb from '@aws-cdk/aws-dynamodb';
import * as cdk from '@aws-cdk/core';
import * as ssm from '@aws-cdk/aws-ssm';

export class DynamoStack extends cdk.Stack {
  regions: string = process.env.REGIONS || 'ap-southeast-2,eu-west-1'
  tableName: string = process.env.APP_NAME!

  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // The code that defines your stack goes here

    const globalTable: dynamodb.Table = new dynamodb.Table(this, this.tableName, {
      partitionKey: {
        name: 'tenant',
        type: dynamodb.AttributeType.STRING
      },
      replicationRegions: this.regions.split(',')
    });

    const tableNameParameter: ssm.StringParameter = new ssm.StringParameter(this, 'tableNameParameter', {
        parameterName: '/shared/MULTI_REGION_ROUTER_TABLE_NAME',
        stringValue: globalTable.tableName
    });

    const tableArnParameter: ssm.StringParameter = new ssm.StringParameter(this, 'tableArnParameter', {
        parameterName: '/shared/MULTI_REGION_ROUTER_TABLE_ARN',
        stringValue: globalTable.tableArn
    });
  }
}

[onv]

@luiseduardocolon, I hope you should have some news regarding GlobalTables CFN support? It was planned to be released month ago. Do you have any changes on your roadmap?

[Ephesoft-Stitus]

@luiseduardocolon - Any chance we can get an update on this? We've been waiting since October for it to be released 'any day'. This is now becoming a real problem for our team.

Thanks for your help,

--Shaun

[stewartcampbell]

I've not seen an announcement for it yet but this just showed up https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dynamodb-globaltable.html

Important You cannot convert a resource of type AWS::DynamoDB::Table into a resource of type AWS::DynamoDB::GlobalTable by changing its type in your template. Doing so might result in the deletion of your DynamoDB table.

[stewartcampbell]

And here's the announcement https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-dynamodb-global-tables-now-support-aws-cloudformation/

[mbarneyjr]

I'm curious what the design decision was in making it a different resource type. Now if anyone wants to go from a single-region to a multi-region architecture, they need to create a brand new AWS::DynamoDB::GlobalTable resource and manage a data migration from the old AWS::DynamoDB::Table resource to the new AWS::DynamoDB::GlobalTable resource

The API lets you update an existing table to a global table, it's unfortunate that CloudFormation doesn't provide the same functionality

[Ephesoft-Stitus]

@mbarneyjr - I'm in that exact position where I have existing tables in a single region and I'd like to create global tables from them. I'm trying to find out if it is supported to manually convert the tables to global tables via the CLI and then import them as new resources into a new stack.

[benbridts]

@Ephesoft-Stitus It's not listed on https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import-supported-resources.html (yet?), but since it's a new resource, the chance of it actually being supported is pretty high