search

crate / cratedb-toolkit

CrateDB Toolkit, an SDK for CrateDB and CrateDB Cloud.
https://cratedb-toolkit.readthedocs.io/
GNU Affero General Public License v3.0
7 stars 3 forks source link

DynamoDB: Properly loading data from tables using single-table design #232

Open amotl opened 3 weeks ago

amotl commented 3 weeks ago

About

Loading data from DynamoDB into CrateDB may need special treatment compared with other databases, because DynamoDB's relational design paradigms are special.

Details

According to the blog post Single-table vs. multi-table design in Amazon DynamoDB:

For people learning about Amazon DynamoDB, the idea of single-table design is one of the most mind-bending concepts out there. Rather than the relational notion of having a table per entity, DynamoDB tables often include multiple different entities in a single table.

DynamoDB isn’t a relational database and you shouldn’t use it like one. The learning curve feels steep, but there are really only three or four key concepts you need to learn, and everything else flows from that. Once you understand these basics, you’ll be able to make more informed decisions about how many tables to use in your application.

Thoughts

Based on reports by others

every row has a unique schema based on the "type" of the record

we think it might make sense to dispatch records to different target tables in CrateDB instead, based on this information.

amotl commented 3 weeks ago

It might make sense to dispatch records to different target tables in CrateDB, based on the "type" of a record in DynamoDB.

@wierdvanderhaar, @hlcianfagna, @hammerhead, @zolbatar: Based on your needs, do you think we need this feature better sooner than later?

amotl commented 3 weeks ago

Status: This is currently not needed too much immediately. Interested people plan to run the data transfer procedure differently, or intend to keep all data in one table.