search

willhlaw / node-firestore-backup-restore

Google Firebase Firestore backup and restore tool
91 stars 24 forks source link

fix for mac: too many opend files & refactor #20

Closed sean-nicholas closed 2 years ago

sean-nicholas commented 6 years ago

Problem

I have a firstore with more than 20.000 documents. If I want to restore them I get the following error after some time Auth error:Error: connect ENFILE 172.217.21.202:443 - Local (undefined:undefined). This seems to stem from the fact that too many files are opened and macOS stops this.

Another problem was that the node process was eating up memory (> 1GB).

Solution

I refactored the code, so that it gets all files at first, puts them in chunks of 20 and then runs the chunks in serial. The 20 documents in one chunk are saved in parallel.

The new code runs pretty fast (3 mins for ~ 20.000 docs) and uses less memory (100 - 120 MB)

Open Qustions

I don't know what chunk size is appropriate. 20 was a guess. Maybe more makes sense? Is there a limit from firebase?

willhlaw commented 6 years ago

Great idea! A lot of people will benefit from this improvement. I'm looking into your changes.

sean-nicholas commented 2 years ago

I'll close this as it pollutes my PR overview :)