How this works to transfer arount 5TB between database?

lihaibh / ditto

NPM library that helps managing MongoDB snapshots easily and efficiently

https://www.npmjs.com/package/mongodb-snapshot

MIT License

50 stars 6 forks source link

How this works to transfer arount 5TB between database? #37

Open andreissf opened 1 year ago

andreissf commented 1 year ago

Hi there, I am looking for an option to restore a datatabase withou losing time generating a file. Has someone tryed that kind of transger with a huge data? I am facing an issue using mongorestore then I am looking for other options.

Regards, Andrei

lihaibh commented 1 year ago

Well I've tested it on a big database (around 10-20 gb) to be honest I didn't try on this scale.

Compare to other solutions, my solution uses the built in Mongodb cursor to iterate through the data and stream it to the other data source, so it should be efficient enough to work in this kind of scale, it will just take time, but you can calculate the percent remain during the transfer.

Pay attention that this is not reversible, while you start streaming the data you cannot roll it back.