Ability to specify maximum number of requests to make in parallel

steadyequipment / node-firestore-backup

Google Firebase Firestore backup tool

190 stars 51 forks source link

Ability to specify maximum number of requests to make in parallel #16

Closed balupton closed 6 years ago

balupton commented 6 years ago

I have about 20 million documents in the firestore to export - at the current rate the exporter seems it exports about 2 a second.

Is it possible for the document downloads to happen in a batch in parallel, so like 20 at a time? As the one-at-a-time serial mode seem to be the bottleneck here.

yoiang commented 6 years ago

Great idea! Feel free to make a PR, not sure how quickly I'll be able to implement this.

balupton commented 6 years ago

ok let me see what I can do

jeremylorino commented 6 years ago

Just ran a test using bluebird Promise.map

test set of:

2 root collections containing 8 "level0" documents each
each "level0" document has between 3 and 4 subcollections itself
each subcollection contains a varying amount of "level1" documents ranging from 0 to 12,000

backupDocument with a concurrency of:

1 - backuptime: 479421.435 ms = ~8 mins
3 - backuptime: 232267.139 ms = ~3.8 mins
- ~48% reduction of time
20 - backuptime: 238634.216 ms = ~3.9 mins

backupDocument with a concurrency of 3 and backupCollection with a concurrency of:

3 - backuptime: 87571.663 ms = ~1.4 mins
- ~81% reduction of time

Thoughts on using bluebird?

balupton commented 6 years ago

@jeremylorino what is that in comparison to?

Is it in comparison the sync version, or the async version at https://github.com/steadyequipment/node-firestore-backup/pull/17

jeremylorino commented 6 years ago

Sorry, I define "concurrency of 1" by using the original code that uses the function promiseSerial.

jeremylorino commented 6 years ago

FYI the total amount of data that was backed up in all cases was 195MB

yoiang commented 6 years ago

@jeremylorino using bluebird is fine! This tasks is interested in providing any async option, which I think is @balupton 's point. It would be great to know the comparison of Javascript Promises to Bluebird promises! I commented in the PR, it would be great if we could make Bluebird (and other future Promise libraries) optional and provide said data for users to decide.

I also appreciate that you did a diminishing returns analysis! Do you know what became the limiting factor? It would be good to know if that is something specific to a situation and whether we should leave it up to the user to analyze decide.

jeremylorino commented 6 years ago

Sweeeet.

I will take a look at the comments on the PR and answer your demising returns questions when I have a chance later today.

Thanks for the input!

jeremylorino commented 6 years ago

@yoiang the latest release closes this. Correct?