search

nasa / RHEAS

Regional Hydrologic Extremes Assessment System
MIT License
73 stars 54 forks source link

Status of TRMM, CHIRPS, and CMORPH downloads #46

Closed AdamJDuncan closed 8 years ago

AdamJDuncan commented 8 years ago

Just an update with a request to see if something can be done.

CHIRPS comes from UCSB. They appear to be temporarily blocking IP addresses in SE Asia after a set amount of data has been downloaded. This lasts for a week or two. Last time I had to download at a cafe. I contacted the data guy nearly two weeks ago but haven't received a reply. This is too bad as it's the longest continuous stretch of precipitation data we have and is thus useful for drought indices. I'll try the other contact.

CMORPH and TRMM come from IRI/Columbia. They will now only download two dates. Like CHIRPS, this is an issue for both Susantha and I. Unlike CHIRPS, it's not an issue of blocking -- I emailed the data contact and he assured me this wasn't the case.

When I first downloaded the full CMORPH set, Narendra recommended I only download four months at a time as there were issues with the script cutting out when it had to jump servers. I tested for longer and shorter periods and found he was correct. My best guess is that the arrangement of data on IRI's servers has recently changed, causing this new issue (though note that it's always two days -- doesn't matter if the Julian date is odd or even).

Best I can do is make a Python script to continue loading dates into the INI file and running --update on loop. Is there a way to insert a workaround into the program? Thanks.

kandread commented 8 years ago

UCSB servers haven't been as reliable as I'd like (although things have improved recently it seems). Unfortunately there's nothing we can do if they block IPs (seems like a strange thing to do). Regarding CMORPH and TRMM, this shouldn't happen. The way it works is that it connects only once to the OpenDAP server and then downloads everything it needs based on the dates requested. Can you post your config here so I can test it out?

AdamJDuncan commented 8 years ago

OK apologies I hadn't tried it after the most recent update -- I think one of your edits inadvertently fixed the issue. CMORPH & TRMM are working now; closing.