search

mchaynes / geodatadownloader

A downloader of geodata
https://geodatadownloader.com
MIT License
70 stars 13 forks source link

Concurrent requests are truly concurrent now #8

Closed mchaynes closed 2 years ago

mchaynes commented 2 years ago

Instead of batching up a "chunk" of requests and waiting for them all to resolve, this instead sets up numConcurrent "workers" via fastq that will start on the next page as soon as a worker is free to do so. This should avoid the problem of sending many fetches all at once, as well as generally speeding up operation

netlify[bot] commented 2 years ago

✔️ Deploy Preview for geodatadownloader ready!

🔨 Explore the source changes: f8bd220039618c6560da1edc6f90dbd71f6032b7

🔍 Inspect the deploy log: https://app.netlify.com/sites/geodatadownloader/deploys/61f0664036342c00083511e5

😎 Browse the preview: https://deploy-preview-8--geodatadownloader.netlify.app/