search

ufs-community / ufs-mrweather-app

UFS Medium-Range Weather Application
Other
23 stars 23 forks source link

NCDC GRIB2 datasets for initializing App #79

Closed ligiabernardet closed 4 years ago

ligiabernardet commented 4 years ago

My understanding is that will support three sources of ICs for initializing the App, always GRIB2 GFS analyses. What is the grid resolution(s) of the data we are supporting (e.g., 0.5 degree? 0.25 degree? multiple options?)

1) Canned 2019-09-09 00 case. The CIME workflow can retrieve this. 2) NOMADS (last 10 days of GFS analyses are available). The CIME workflow can retrieve this. 3) NCDC archives (CIME cannot retrieve this now, but users can download and stage the data. CIME could be augment to retrieve this). Can someone let me know what is the URL for this data? For example, is it the 0.5 deg at ftp://nomads.ncdc.noaa.gov/GFS/analysis_only?

In addition, can users with access to NOAA platforms get the ICs from NOAA HPSS? If so, can someone tell me where?

Thanks, Ligia

ligiabernardet commented 4 years ago

@llpcarson @LarissaReames-NOAA @uturuncoglu @arunchawla-NOAA

LarissaReames-NOAA commented 4 years ago

The main page I used when retrieving archived data from NCEI (formerly NCDC) is :

https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/global-forcast-system-gfs

This has various ways listed to access the data (https, FTP, HAS, etc.). Online GFS4 (0.5 deg) data is only available back to the January 2019:

https://nomads.ncdc.noaa.gov/data/gfs4/

But data back to 2006 can be requested from HAS:

https://www.ncdc.noaa.gov/has/HAS.FileAppRouter?datasetname=GFSGRB24&subqueryby=STATION&applname=&outdest=FILE

LarissaReames-NOAA commented 4 years ago

Also, I believe the plan was to support 0.25, 0.5, and 1.0 degree.

uturuncoglu commented 4 years ago

Canned 2019-09-09 00 case. The CIME workflow can retrieve this.

Yes. The data is retrieved from NOMADS server and it process gfs.t00z.atmanl.nemsio and gfs.t00z.sfcanl.nemsio files using CHGRES to create input.

NOMADS (last 10 days of GFS analyses are available). The CIME workflow can retrieve this.

If the data is not in the input folder. CIME gets the data from https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/ and process it same way with (1). Again, we are only supporting gfs.t00z.*anl.nemsio files at this point.

NCDC archives (CIME cannot retrieve this now, but users can download and stage the data. CIME could be augment to retrieve this). Can someone let me know what is the URL for this data? For example, is it the 0.5 deg at ftp://nomads.ncdc.noaa.gov/GFS/analysis_only?

I think there is no specific namelist variable for chgres for the input data resolution. Right? It is also possible to add support for archived files using

https://nomads.ncdc.noaa.gov/data/gfsanl/

but in here the data format is grb2 (I know that CHGRES supports it now) and I think we need to make some change in the namelist file. Please let me know the example namelist file to process this data and I could make required changes in the CIME side to automatically generate it and test it.

arunchawla-NOAA commented 4 years ago

please wait till the library build and ufs-utils release branches are updated for that. We plan to test grib2 data next week

Arun Chawla Chief Engineering & Implementation Branch Room 2083 National Center for Weather & Climate Prediction 5830 University Research Court College Park, MD 20740 Phone : 301-683-3740 Mobile : 240-564-5675 Fax : 301-683-3703

On Fri, Jan 31, 2020 at 3:31 PM Ufuk Turunçoğlu notifications@github.com wrote:

Canned 2019-09-09 00 case. The CIME workflow can retrieve this.

Yes. The data is retrieved from NOMADS server and it process gfs.t00z.atmanl.nemsio and gfs.t00z.sfcanl.nemsio files using CHGRES to create input.

NOMADS (last 10 days of GFS analyses are available). The CIME workflow can retrieve this.

If the data is not in the input folder. CIME gets the data from https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/ and process it same way with (1). Again, we are only supporting gfs.t00z.*anl.nemsio files at this point.

NCDC archives (CIME cannot retrieve this now, but users can download and stage the data. CIME could be augment to retrieve this). Can someone let me know what is the URL for this data? For example, is it the 0.5 deg at ftp://nomads.ncdc.noaa.gov/GFS/analysis_only?

I think there is no specific namelist variable for chgres for the input data resolution. Right? It is also possible to add support for archived files using

https://nomads.ncdc.noaa.gov/data/gfsanl/

but in here the data format is grb2 (I know that CHGRES supports it now) and I think we need to make some change in the namelist file. Please let me know the example namelist file to process this data and I could make required changes in the CIME side to automatically generate it and test it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/79?email_source=notifications&email_token=AL5NYI2JHVLTBW6SORL32LDRASDATA5CNFSM4KOLW4P2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKP5GEA#issuecomment-580899600, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL5NYIYEMA7UYNQ5SKDKCRTRASDATANCNFSM4KOLW4PQ .

uturuncoglu commented 4 years ago

@arunchawla-NOAA Sure. Just let me know, when it is ready.

ligiabernardet commented 4 years ago

Thank you for the information. It has been captured in the App UG and later we can review it.

arunchawla-NOAA commented 4 years ago

@ligiabernardet can we close this ?

ligiabernardet commented 4 years ago

We need to decide if CIME will be augmented to retrieve data from NCEI archives. Per Ufuk: CIME cannot retrieve this now, but users can download and stage the data. CIME could be augment to retrieve this. If CIME will not be augmented, we need to understand where/how a user should stage the data on disk so CIME can find it. And test it. The documentation is currently incomplete in this aspect.

On Wed, Feb 5, 2020 at 9:31 AM arun chawla notifications@github.com wrote:

@ligiabernardet https://github.com/ligiabernardet can we close this ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/79?email_source=notifications&email_token=AE7WQAT6R3IGGSWHQQYKIXLRBLSV5A5CNFSM4KOLW4P2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK4CG7A#issuecomment-582493052, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7WQAWSAJ7RVIJZJE7EYSDRBLSV5ANCNFSM4KOLW4PQ .

rsdunlapiv commented 4 years ago

@uturuncoglu What would be required to add the NCEI NOMADS download option to CIME?

Per @LarissaReames-NOAA :

This has various ways listed to access the data (https, FTP, HAS, etc.). Online GFS4 (0.5 deg) data is only available back to the January 2019:

https://nomads.ncdc.noaa.gov/data/gfs4/

mvertens commented 4 years ago

This is already being supported in CIME for NEMSio files. where do we get GRIB?

On Thu, Feb 6, 2020 at 9:08 AM Rocky Dunlap notifications@github.com wrote:

@uturuncoglu https://github.com/uturuncoglu What would be required to add the NCEI NOMADS download option to CIME?

Per @LarissaReames-NOAA https://github.com/LarissaReames-NOAA :

This has various ways listed to access the data (https, FTP, HAS, etc.). Online GFS4 (0.5 deg) data is only available back to the January 2019:

https://nomads.ncdc.noaa.gov/data/gfs4/

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/79?email_source=notifications&email_token=AB4XCE53F52KLYR7NVWF5ALRBQYYTA5CNFSM4KOLW4P2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK7YYUA#issuecomment-582978640, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4XCE5JG4HQCUTEORJ77I3RBQYYTANCNFSM4KOLW4PQ .

uturuncoglu commented 4 years ago

@mvertens @rsdunlapiv We need,

rsdunlapiv commented 4 years ago

@uturuncoglu does this link provided by @LarissaReames-NOAA work? https://nomads.ncdc.noaa.gov/data/gfs4/

uturuncoglu commented 4 years ago

It has data but it goes back to 2019 and I did not try to process it. We need an example chores_cube namelist file to test chgres standalone.

uturuncoglu commented 4 years ago

I could able to process grib2 data with chgres and now I'll make required changes in the CIME side to make it automatic and test the model.

@KateFriedman-NOAA If you don't mind could you put UFS_UTILS/parm/varmap_tables/GFSphys_var_map.txt file to the FTP.

KateFriedman-NOAA commented 4 years ago

@uturuncoglu Sure, I just need to know which UFS_UTILS tag/branch to grab that file from. @GeorgeGayno-NOAA Can you point me to the UFS_UTILS copy for the release? Thanks!

GeorgeGayno-NOAA commented 4 years ago

I could able to process grib2 data with chgres and now I'll make required changes in the CIME side to make it automatic and test the model.

@KateFriedman-NOAA If you don't mind could you put UFS_UTILS/parm/varmap_tables/GFSphys_var_map.txt file to the FTP.

The GFSphys_var_map.txt file is part of the release branch: https://github.com/NOAA-EMC/UFS_UTILS/tree/release/public-v1/parm/varmap_tables

KateFriedman-NOAA commented 4 years ago

Thanks @GeorgeGayno-NOAA ! @uturuncoglu You can now find that parm file here:

https://ftp.emc.ncep.noaa.gov/EIB/UFS/parm/GFSphys_var_map.txt

rsdunlapiv commented 4 years ago

Here is an update on CIME support for automatic downloading of initial conditions for chgres:

@jedwards4b and @uturuncoglu should please review and correct any issues with the above.

arunchawla-NOAA commented 4 years ago

My recommendation is to disable the automatic download of grib2 data. While this is an attractive feature it does force you to rely on two things

  1. The data feed being available on said website and
  2. If the website for any reason is blocked at the user end then this will show up as a CIME failure

It may be better to provide detailed documentation on where data can be obtained from and let the user directly interact with website(s) to download the data sets.

arunchawla-NOAA commented 4 years ago

@rsdunlapiv @jedwards4b @GeorgeGayno-NOAA @LarissaReames-NOAA

I checked out the different ways to get data from here

https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/global-forcast-system-gfs

The only complete set was through their HAS system (I was getting 0.5 deg grib2 fields for dorian test case) and that is an interactive approach. Their NOMADS server has holes in the data sets.