search

blaylockbk / Herbie

Download numerical weather prediction datasets (HRRR, RAP, GFS, IFS, etc.) from NOMADS, NODD partners (Amazon, Google, Microsoft), ECMWF open data, and the University of Utah Pando Archive System.
https://herbie.readthedocs.io/
MIT License
503 stars 75 forks source link

Go to next source option if there is a connection issue #246

Open williamhobbs opened 11 months ago

williamhobbs commented 11 months ago

Herbie works great on my home network. The firewall on my office/corporate network, however, often causes issues with certificates that result in an SSLError. For other projects (e.g., getting data from the NREL NSRDB), I can usually get around this by using a local copy of a certificate. I have not been able to make that work with Herbie and AWS.

So, I ended up stuck with errors like:

SSLError: HTTPSConnectionPool(host='noaa-hrrr-bdp-pds.s3.amazonaws.com', port=443): Max retries exceeded with url: /hrrr.20210701/conus/hrrr.t12z.wrfsfcf06.grib2 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))

I just had the idea to test Herbie with priority='azure' and it works just fine ('google' works as well).

Would it make sense for Herbie to automatically cycle through the other source options if it runs into this type of error?

blaylockbk commented 11 months ago

This would be a good feature to add. Just need to know how to check for this error before moving on.

In the mean time, I believe you can change the source priority in the Herbie config file ~/.config/herbie/config.toml

  [default]
  model = "hrrr"
  fxx = 0
  save_dir = "~/data"
  overwrite = false
  verbose = true
+ priority = ['azure', 'google', 'aws', 'nomads']
yifang-zhu-pge commented 2 weeks ago

Same here. AWS gives me a SSLError too. It worked after I changed "google" as the first option.