edasque / DynamoDBtoCSV

Dump DynamoDB data into a CSV file
Apache License 2.0
471 stars 152 forks source link

Revert "Fixes the issue where large data sets cause a heap error." #46

Closed smolse closed 6 years ago

smolse commented 6 years ago

This reverts commit 3afcb35b554f8f8abe75d77ffde5e8d70d95790b.

That commit introduced a bug when order of columns differ between subsequent console prints and this results in completely invalid CSV files having values mixed up. Moreover, the commit does an assumption that all attributes are returned in 1000 first items which might not be always true. Heap error needs to be fixed by possibly dumping interim results to a file or in some other way, e.g. producing multiple partial CSV files, which then could be merged using other tool, but logic anyway should be free of any tricky assumptions.