wholmgren
commented
3 years ago also related: https://github.com/SolarArbiter/solarforecastarbiter-core/issues/680 #674
Open williamhobbs opened 3 years ago
wholmgren
commented
3 years ago also related: https://github.com/SolarArbiter/solarforecastarbiter-core/issues/680 #674
williamhobbs
commented
2 years ago Adding a note here in case it is useful to someone: I've found that when I need to catch up on fetching NWP files (e.g., when my SD card fills up and I don't notice for a few days and I have to clean things up and get re-started), adding a 120 second pause between fetchnwp --once commands for HRRR subhourly files seems to work well at avoiding NOMADS rate limiting.
In Raspberry PI OS terminal, I run something like:
x=45; for ((n=0; n < (x / 2); n++)); do solararbiter${IFS}fetchnwp${IFS}-v${IFS}--once${IFS}/home/pi/Downloads/sfa${IFS}hrrr_subhourly; sleep 120; done
This repeats the command solararbiter fetchnwp -v --once /home/pi/Downloads/sfa hrrr_subhourly 45 times, pausing 120 seconds between each run (to try to avoid NOAA download rate limits). The ${IFS} adds necessary spaces to the command.
This typically returns 19 .grib2 files, so it may be that about 10 files per minute is below the threshold for getting limited.
Edit: I'm not sure what the "x / 2" was for. The command should have been:
x=45; for ((n=0; n < x; n++)); do solararbiter${IFS}fetchnwp${IFS}-v${IFS}--once${IFS}/home/pi/Downloads/sfa${IFS}hrrr_subhourly; sleep 120; done
williamhobbs
commented
2 years ago An additional scenario has come up where this could be useful: following NOMADS outages. On 2022-08-15, portions of NOMADS were down for about 6 hours. Estimated time to full repair was approximately 48 hours.
If I make too many requests to NOMADS, it will start returning 0 kB .grib2 files. In some cases,
solararbiter fetchnwpdoes not seem to notice this and proceeds to create a .nc file that is missing data. It seems like having a way to eitherwould be helpful.
Details on an example test case are below, where I tried fetching gefs while also fetching hrrr_subhourly. The easy solution here is that I shouldn't fetch gefs while running other fetch processes, but it seems possible that this same issue could come up in unavoidable ways, e.g., after a multi-day network outage and
solararbiter fetchnwpcatches up all at once when the network comes back.I have a regular existing process running, `solararbiter fetchnwp -v /home/pi/Downloads/sfa hrrr_subhourly'
I started a test process at ~ 2021-07-23 15:58 UTC,
solararbiter fetchnwp -v /home/pi/Downloads/sfa gefs, and quickly started being throttled by NOAA, receiving 0 kB files'fetchnwp' had already started on
hrrr_subhourly. This resulted in getting 0 kB hrrr files being downloaded:ls -l output part of the way through fetching (note the empty files for f04-f06):
ls -l output after convertsion to NetCDF finished (~18 MB .nc file):
fetchnwp verbose output (warnings issued for f03, f07, f08, but not the empty files f04-f06):
after deleting the folder, running
solararbiter fetchnwp -v --once /home/pi/Downloads/sfa hrrr_subhourlyresulted in all new files (none 0 kB):