search

markitx / dynamo-backup-to-s3

Stream DynamoDB backups to S3
MIT License
141 stars 106 forks source link

Missing items in backup #20

Closed nathanwelch closed 8 years ago

nathanwelch commented 8 years ago

Hi I have a table with ~1200 items and when I back it up I used to only get about half of them on version 0.1.9. I updated to the latest version and now I get more but still not all. Interestingly, the percentage of records backed up goes down as I increase the read throughput. For example, I get ~850 when I have read percentage at 75% but ~1100 at 25%.

I did some initial debugging by modifying the code to emit an event for every item that's processed and I can confirm that all items are processed. Additionally if I modify dynamo-backup.js to save all records to a file and then upload that file, I can confirm they are all processed and uploaded. To me this points to some bug either in the readable stream or in what is handling the S3 upload stream. Given that the percentage of items backed up goes up as the read rate goes down, I'm guessing it's reading from dynamo faster than it can write to S3 and that the readable stream cannot handle this.

Is anyone aware of any issues that would cause this? My temporary fix in my fork (https://github.com/markitx/dynamo-backup-to-s3/compare/master...smashgg:backup-to-file) is to add an option to backup to file and then upload that file after the dynamo backup has completed.

Node version: Tried in 0.10.x, 0.12.x, and 6.2.2 with fresh installs between each version change dynamo-backup-to-s3 version: 0.4.x

Smbc1 commented 8 years ago

+1, our backups size variates day after day: 25GB, 6GB, 25GB for the same table. No more facts about it, still investigating. Looks like AWS has changed something about week or two ago. I had no such problems before.

dylanlingelbach commented 8 years ago

This is very odd. I'll try to test some tomorrow. I haven't been using this project lately so difficult for me to test. Let me know if anyone gets more data here

dylanlingelbach commented 8 years ago

Is anyone still experiencing this issue?

dylanlingelbach commented 8 years ago

Going to close this for now. Reopen if you are still experiencing this