search

worldbank / blackmarblepy

Georeferenced Rasters and Statistics of Nightlights from NASA Black Marble
https://worldbank.github.io/blackmarblepy/
Mozilla Public License 2.0
26 stars 4 forks source link

Issues when downloading many files #40

Open ramarty opened 6 months ago

ramarty commented 6 months ago

I was downloading data for multiple days and the "retrieving" failed a few times. The function still worked, as it's designed to retry. But not sure if there's something going on where it's trying to download too much at one time? Not sure if limiting to, say, 2-3 concurrent downloads would help this issue? (I never seem to get these red bars when downloading just a couple tiles at once).

Screen Shot 2023-12-20 at 6 43 16 PM

ramarty commented 6 months ago

My internet was really bad when this happened - when I switched to better internet it didn't happen! But maybe still something to think about?

Holly-Transport commented 6 months ago

I also encountered many instances of file download timing out, like this (I used a smaller country, Jamaica): image

g4brielvs commented 6 months ago

I was downloading data for multiple days and the "retrieving" failed a few times. The function still worked, as it's designed to retry. But not sure if there's something going on where it's trying to download too much at one time? Not sure if limiting to, say, 2-3 concurrent downloads would help this issue?

@ramarty @Holly-Transport Thanks (and happy new year)!

That's something to consider. There seems to be 2 things at play. The number of parallel downloads may be too ambitious. In the next release, we can change the default and expose the parameter for the user to choose. Also, especially in slower connections, the default httpx.Client may be timing out too soon. This issue would persist even if we decreased the number of parallel downloads. We can increase the default timeout and also expose it to the user.

All in all, connection issues will always be an upstream constraint imposed by NASA LAADS DAAC, but we can try to find the sweet spot. What do you think?