search

edasque / DynamoDBtoCSV

Dump DynamoDB data into a CSV file
Apache License 2.0
471 stars 152 forks source link

Out of memory on huge amount of records #28

Closed rkoblev closed 5 years ago

rkoblev commented 7 years ago

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

It happens on huge amount of records. In my case about 4 million records

edasque commented 7 years ago

Sorry, can't help there, I don't have a DynamoDB db anymore.

tcchau commented 6 years ago

@rkoblev This happens because the change I made to deal with inconsistent "schema" in the table means that the code has to look at all the data first before being able to determine what the headers are. I can create a mode of operation for when you have a fixed schema that can avoid the large database problem, if you're still interested in this. Please write back if you are.

rkoblev commented 6 years ago

@tcchau Thanks, but I'm not interested anymore. It was one time job

sanjay-k7r commented 6 years ago

Got the same error as @rkoblev with around 800K records. @tcchau - It would be good to have another mode of operation as you suggest.

thirstyfish commented 6 years ago

Same issue here

Zhang21 commented 5 years ago

same problem：FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Zhang21 commented 5 years ago

The method is increase node limit memory, because default limit memory of node is not enough.

# 2GB(2048MB)
node --max-old-space-size=2048 xxx.js