search

SolarArbiter / solarforecastarbiter-core

Core data gathering, validation, processing, and reporting package for the Solar Forecast Arbiter
https://solarforecastarbiter-core.readthedocs.io
MIT License
33 stars 21 forks source link

update fetch.nwp for new GFS directory structure #662

Closed sentry-io[bot] closed 3 years ago

sentry-io[bot] commented 3 years ago

From GFS upgrade document...

Effective on or about March 22, 2021, beginning with the 1200 Coordinated Universal Time (UTC) run, the National Centers for Environmental Prediction (NCEP) will upgrade the GFS and Global Data Assimilation System (GDAS) from version 15.3 to 16.0. 

...

1. Changes to directory structures on the NCEP Web services
WAVEWATCH III multi_1 products found on NOMADS/ftpprd in: pub/data/nccf/com/wave/prod/
will be replaced by GFS.v16 Wave products and moved to the GFS directory structure for NOMADS/ftpprd:
       pub/data/nccf/com/gfs/prod/
The directory structure for all GFS products on NOMADS/ftpprd will be changed to differentiate products from the atmospheric and wave components, respectively.
Atmospheric output: pub/data/nccf/com/gfs/prod/gfs.YYYYMMDD/CC/atmos pub/data/nccf/com/gfs/prod/gdas.YYYYMMDD/CC/atmos pub/data/nccf/com/gfs/prod/enkfgdas.YYYYMMDD/CC/atmos
Wave output: pub/data/nccf/com/gfs/prod/gfs.YYYYMMDD/CC/wave pub/data/nccf/com/gfs/prod/gdas.YYYYMMDD/CC/wave

Indeed, the fetch hangs after the 2021/03/22/06 run:

2021-03-26 11:28:56,892 INFO Getting file /tmp/tmp427mltu3/2021/03/22/06/gfs.t06z.pgrb2.0p25.f384.grib2
2021-03-26 11:29:04,253 INFO Converting GRIB files to NetCDF with wgrib2
2021-03-26 11:29:21,060 INFO Optimizing NetCDF file to save at /home/will/git_repos/solarforecastarbiter-core/gfs_0p25/2021/03/22/06/gfs_0p25.nc
2021-03-26 11:29:22,736 INFO Done optimizing NetCDF at /home/will/git_repos/solarforecastarbiter-core/gfs_0p25/2021/03/22/06/gfs_0p25.nc
2021-03-26 11:29:22,752 INFO Moving on to next model run
^C%                                                                                                                                                                                                                                           (sfacore) 11:33:26 will@golem.energy.arizona.edu solarforecastarbiter-core master ? solararbiter fetchnwp . gfs_0p25 --use-tmp -v
2021-03-26 11:34:13,388 INFO Fetching NWP forecasts for gfs_0p25

I tried adding /atmos to GFS_0P25_1HR['dir'], but the fetch script still hangs.

Sentry Issue: WORKER-API-Q

Could not process group of 04f7dd70-f83b-11ea-922a-0a580a8003c2, /data/gfs_0p25/2021/03/26/00/gfs_0p25.nc does not exist
wholmgren commented 3 years ago

I tried adding /atmos to GFS_0P25_1HR['dir'], but the fetch script still hangs.

Correction, this does actually work. Also with -vv I now understand that it hangs because it's waiting for the file to be available. 🤦 🤦