search

crate / cratedb-toolkit

CrateDB Toolkit, an SDK for CrateDB and CrateDB Cloud.
https://cratedb-toolkit.readthedocs.io/
GNU Affero General Public License v3.0
7 stars 3 forks source link

DynamoDB: Add pagination support for `full-load` table loader #252

Closed amotl closed 2 months ago

amotl commented 2 months ago

About

For making DynamoDB full-load operations more efficient, use CrateDB's bulk operations. With this patch, the batching towards CrateDB is implemented by doing pagination on the DynamoDB source database. For other strategies, see backlog.

References

Backlog

For subsequent iterations:

/cc @wierdvanderhaar, @hlcianfagna