search

worldbank / blackmarblepy

Georeferenced Rasters and Statistics of Nightlights from NASA Black Marble
https://worldbank.github.io/blackmarblepy/
Mozilla Public License 2.0
26 stars 4 forks source link

Read Operation Timeout #62

Open eflplaton opened 4 months ago

eflplaton commented 4 months ago

Read Operation Timeout

Description

I tried running the code before and it works perfectly even though it took time to process a whole year's worth of data, hence, I divided the code by month. Now, I tried to rerun it and tried several ways like adjusting the timeout, and the size into a day, but it just usually takes hours until it stops progressing and stays into the same % of "PROCESSING TASKS", is there a way to manage this error? Moreover, the file_skip_if_exists doesn't work since I tried running the same query hoping it will skip the tile with available data, but it just resets the whole process.

Reproducibility

Screenshots / Error Messages (if applicable)

image

Here is the screenshot where I ran the same code and it supposedly skipped the file since before it had xKB of size, now it resets to: image

Environment

g4brielvs commented 4 months ago

@eflplaton Thanks so much for reporting this issue. It seems to be related to known issues #38 and #40. Fortunately, #38 will be addressed in the next release. On #40, please note that BlackMarblePy retrieves the NASA LAADS DAAC - thus it depends on NASA's servers. It will automatically retry failed connections, but after too many requests, you may experience successive errors. For resolution in the next release, I'll try adjusting and reducing the default number of parallel connections. We would greatly appreciate your patience and keep please stay tuned for updates.

Also, if you're looking for monthly composites, I'd suggest using VPN46A3 instead. In fact, we could use both as in this example.

ramarty commented 4 months ago

@g4brielvs Makes sense on adjusting number of default number of parallel connections! Wonder if makes sense to have a default, but also add a parameter so users can increase/decrease? For example, if time is less of an issue & want to avoid making too many requests at a time, this would allow setting to parallel connections to 1 in those cases.

eflplaton commented 4 months ago

Hello, @g4brielvs and @ramarty! I sincerely appreciate your prompt response and the effort you're putting into addressing the issue despite your busy schedule. For the next release addressing issue #38, if it is possible to share, could you kindly give me an estimate on when we might expect the next release? (no pressure intended ☺️)

Thank you once again for your support and understanding.

g4brielvs commented 4 months ago

@eflplaton @ramarty When trying to download from NASA LAADS DAAC directly, I'm encountering connection failures as well - which leads me to believe this is an upstream issue. There is scheduled system reboot, that may or may not be related. Are you running into the same issue?

eflplaton commented 4 months ago

@eflplaton @ramarty When trying to download from NASA LAADS DAAC directly, I'm encountering connection failures as well - which leads me to believe this is an upstream issue. There is scheduled system reboot, that may or may not be related. Are you running into the same issue?

Hello @g4brielvs, yes I run into the same issue. I even tried VNP46A3 but it's still in 0% after two days.