Add support to resume process in case of crashes

amazon-archives / dynamodb-import-export-tool

Exports DynamoDB items via parallel scan into a blocking queue, then consumes the queue and import DynamoDB items into a replica table using asynchronous writes.

Apache License 2.0

90 stars 38 forks source link

Add support to resume process in case of crashes #5

Open marcosnils opened 8 years ago

marcosnils commented 8 years ago

I need to scan a huge collection and sometimes my process crashes due to OOM. It'd be nice to save the LastEvaluatedKey of each process (in a file maybe?) so if I start the process again I can indicate to resume the scan instead of starting all over

thoughts?

amcp commented 7 years ago

@marcosnils This could be implemented with a checkpoint table for a completely automated copy recovery solution. However, I do recommend you tune your execution parameters so that you avoid OOM as well (see my reply in #2). Pull requests are welcome.