search

ufs-community / ufs-mrweather-app

UFS Medium-Range Weather Application
Other
23 stars 23 forks source link

We need a plan for connecting CIME to the public input data sources #15

Closed mvertens closed 4 years ago

mvertens commented 4 years ago

In order for CIME to be extended to support different forecast initial conditions flexibly, we need to understand the directory structure in the FTP site that will be made available. We need to understand the requirements for CIME to obtain this data and the list of files that will be needed. A major concern is the potentially large size of some of the files - and we need to determine if they need to be manually downloaded or have CIME download them automatically as part of its workflow.

arunchawla-NOAA commented 4 years ago

We have an ftp site that was opened up for staging data. Documentation is still weak. Suggest a meeting with Jun Wang and Kate Friedman to bolster this. We should explore using GIT-LFS added to mr-weather-app or to the weather-model to see if the data can be downloaded automatically

junwang-noaa commented 4 years ago

@mvertens, could we have a meeting to discuss this issue? Thanks.

rsdunlapiv commented 4 years ago

I will try to put something on the calendar to discuss. I am interested in @arunchawla-NOAA suggestion of GIT-LFS and how that repo could be connected to CIME to download data as needed. Another question is whether there is a standard directory structure expected on supported platforms, e.g., fixed files, ICs, etc. - is there a standard structure already used on NOAA systems, e.g., Hera, that should be generalized and adopted across all platforms. CESM uses this idea of a shared root data directory - how well does this apply here?

rsdunlapiv commented 4 years ago

Could @KateFriedman-NOAA please provide a list of current input data sources expected to be used by the release? e.g., the FTP site and any others, GIT-LFS, etc.

uturuncoglu commented 4 years ago

In the current ftp site the input files (except static ones) are in tar format and it is hard to access individual files without extracting the file. We need to have a hierachical and standardized folder structure rather than having single tar file for input. It could be also nice to have the input files for different resolution (i.e. C384) and also support for CCPP.

rsdunlapiv commented 4 years ago

Given the large size of many of these files - should we support a compression mechanism?

KateFriedman-NOAA commented 4 years ago

I know of the following on the EMC ftp site:

1) FIX files: https://ftp.emc.ncep.noaa.gov/EIB/UFS/global/fix

...which are broken into sub-groups:

This is the full collection of FV3GFS fix files as of November 12th. Most likely overkill. Do we need a paired down collection?

2) C96 canned case tarball (originally made available for NASA folks)

https://ftp.emc.ncep.noaa.gov/EIB/UFS/RT/fv3_gfdlmprad.tar

rsdunlapiv commented 4 years ago

@KateFriedman-NOAA this is great and already very helpful! Can you provide (or is there already) a brief description of what's in each of those subdirectories? Are these subdirs expected to be reproduced as is on all supported platforms?

rsdunlapiv commented 4 years ago

@KateFriedman-NOAA The global/fix directory has the fixed files. What is the public source for users to retrieve initial conditions and boundary conditions? Is the plan to host those separately?

KateFriedman-NOAA commented 4 years ago

@KateFriedman-NOAA this is great and already very helpful! Can you provide (or is there already) a brief description of what's in each of those subdirectories? Are these subdirs expected to be reproduced as is on all supported platforms?

I'm mostly a facilitator of copying these files to our supported platforms so I can't give a detailed description. Is one needed? Here is a very brief description:

When we get updated fix files myself or another developer copies it into one of the main collections (FIX_DIR) on the WCOSS-Dell and then I copy them to the FIX_DIRs on WCOSS-Dell (other side), both WCOSS-Crays, Hera, and Jet. I also save the whole collection in a new tarball on HPSS.

If needed I can copy whatever final set of fix files folks land on for the release to the supported platforms. We hold them under a group account.

@KateFriedman-NOAA The global/fix directory has the fixed files. What is the public source for users to retrieve initial conditions and boundary conditions? Is the plan to host those separately?

There is no public source currently. Our archival server (HPSS) is not accessible by the public, only from NOAA machines. The release team would need to post a sample set online somewhere (our ftp server maybe). NCEI is a public source for model output but I don't see the restart files there, just some post-processed grib output. Access to model output, especially initial conditions, has become a real issue for anyone who doesn't have access to a NOAA machine. :(

ligiabernardet commented 4 years ago

The list of fix files needed for the runs supported in this release is in the draft UFS WM User's Guide. Work is in progress by @lpcarson to add one-line descriptions for each fix file This is in Google Doc https://docs.google.com/document/d/1D7aupwMAjIdv_8eHiRtdkyad_o_f_eGmpGFIvBeZXcg/edit#. It will later be converted to Sphinx. @KateFriedman-NOAA https://github.com/KateFriedman-NOAA if you can contribute to the description these files, please do!

On Tue, Dec 10, 2019 at 8:11 AM Kate Friedman notifications@github.com wrote:

@KateFriedman-NOAA https://github.com/KateFriedman-NOAA this is great and already very helpful! Can you provide (or is there already) a brief description of what's in each of those subdirectories? Are these subdirs expected to be reproduced as is on all supported platforms?

I'm mostly a facilitator of copying these files to our supported platforms so I can't give a detailed description. Is one needed? Here is a very brief description:

  • fix_am - atmospheric
  • fix_chem - chemistry
  • fix_fv3 - fv3
  • fix_fv3_gmted2010 - another fv3
  • fix_gldas - GLDAS
  • fix_orog - orography
  • fix_sfc_climo - surface climo
  • fix_verif - verification

When we get updated fix files myself or another developer copies it into one of the main collections (FIX_DIR) on the WCOSS-Dell and then I copy them to the FIX_DIRs on WCOSS-Dell (other side), both WCOSS-Crays, Hera, and Jet. I also save the whole collection in a new tarball on HPSS.

If needed I can copy whatever final set of fix files folks land on for the release to the supported platforms. We hold them under a group account.

@KateFriedman-NOAA https://github.com/KateFriedman-NOAA The global/fix directory has the fixed files. What is the public source for users to retrieve initial conditions and boundary conditions? Is the plan to host those separately?

There is no public source currently. Our archival server (HPSS) is not accessible by the public, only from NOAA machines. The release team would need to post a sample set online somewhere (our ftp server maybe). NCEI is a public source for model output but I don't see the restart files there, just some post-processed grib output. Access to model output, especially initial conditions, has become a real issue for anyone who doesn't have access to a NOAA machine. :(

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/15?email_source=notifications&email_token=AE7WQARLHSRZSITWGWJ6ZZ3QX6WQNA5CNFSM4JXCWJV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGPSJII#issuecomment-564077729, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7WQAVUM6KSECE6UQY55KTQX6WQNANCNFSM4JXCWJVQ .

mvertens commented 4 years ago

@arunchawla-NOAA @KateFriedman-NOAA @ligiabernardet -

@rsdunlapiv @jedwards4b and @mvertens propose the following structure for managing and hosting input data for the release:

ligiabernardet commented 4 years ago

The proposal about fix files seems good to me.

The proposal for initial files is fine along as it is extensible. It is my understanding that the Data Prep release team is working on having the capability for the MR Weather App to run from GFS GRIB2 files available through public archives (such as NOAA NOMADS and NCAR). This is very important for the community to use this app for research. It would be good if CIME can start the app from those public files. User may have to download and stage the data on disk by hand - that is fine.

Lateral boundary conditions are not needed for the global MR Weather app configuration because it is a global domain.

jedwards4b commented 4 years ago

@ligiabernardet Can you provide an example of files from these public archives that would allow us to run the model? Thanks

ligiabernardet commented 4 years ago

No, I cannot. The current model I have access to does not work from GFS GRIB2 files available in public archives. The UFS release data prep team was doing enhancements to allow this capability. @LarissaReames-NOAA Do you have any update on this?

LarissaReames-NOAA commented 4 years ago

@ligiabernardet @jedwards4b The files that we have been testing against are located on NCEP's http server of the form gfs.tCCz.pgrb2.0p25.fFFF or gfs.tCCz.pgrb2.0p50.fFFF. We're also looking to support older archived files on NCDC's nomads server. This is just one example of those files. Functionally, these files should be very similar.

There is also a NCEI server (Arun).

jedwards4b commented 4 years ago

@LarissaReames-NOAA Do you have a timeline of when I could expect to be able to run the ufs_mrweather_app using files from nomads?

LarissaReames-NOAA commented 4 years ago

@jedwards4b We fixed one last bug in implementing the surface parameter processing code yesterday, so we'll be able to start testing that soon. @arunchawla-NOAA might have more information on how long he thinks that might take.

uturuncoglu commented 4 years ago

@LarissaReames-NOAA is it possible to extract fv3_gfdlmprad.tar under /EIB/UFS/RT directory in the ftp side. By this way, CIME could access individual required files in it.

KateFriedman-NOAA commented 4 years ago

@LarissaReames-NOAA is it possible to extract fv3_gfdlmprad.tar under /EIB/UFS/RT directory in the ftp side. By this way, CIME could access individual required files in it.

@uturuncoglu I put that tarball up on our ftp server and have access to it. @arunchawla-NOAA @junwang-noaa Any objections if I unpack the tarball on our ftp server?

arunchawla-NOAA commented 4 years ago

No it is just a stand alone benchmark test case that people can download and run. However you want to store it.

Arun Chawla Chief Engineering & Implementation Branch Room 2083 National Center for Weather & Climate Prediction 5830 University Research Court College Park, MD 20740 Ph: 301-683-3740 Fx: 301-683-3703

On Thu, Dec 12, 2019 at 9:13 AM Kate Friedman notifications@github.com wrote:

@LarissaReames-NOAA https://github.com/LarissaReames-NOAA is it possible to extract fv3_gfdlmprad.tar https://ftp.emc.ncep.noaa.gov/EIB/UFS/RT/fv3_gfdlmprad.tar under /EIB/UFS/RT directory in the ftp side. By this way, CIME could access individual required files in it.

@uturuncoglu https://github.com/uturuncoglu I put that tarball up on our ftp server and have access to it. @arunchawla-NOAA https://github.com/arunchawla-NOAA @junwang-noaa https://github.com/junwang-noaa Any objections if I unpack the tarball on our ftp server?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/15?email_source=notifications&email_token=AL5NYI75VN6BHXBOEN3QLVLQYJBKLA5CNFSM4JXCWJV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGWZE6Y#issuecomment-565023355, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL5NYI5JSEYFU5GIMHR3U4TQYJBKLANCNFSM4JXCWJVQ .

KateFriedman-NOAA commented 4 years ago

I have unpacked the tarball:

https://ftp.emc.ncep.noaa.gov/EIB/UFS/RT/fv3_gfdlmprad/

The tarball itself is still there under the ../RT folder alongside the unpacked folder.

rsdunlapiv commented 4 years ago

Just so we don't lose this, I wanted to document here that on the call yesterday we decided to version fixed file directories (not per-file versioning, for now) and that @KateFriedman-NOAA would set up the initial versioned structure and pass it along to the CIME team to integrate with its input data mechanism.

KateFriedman-NOAA commented 4 years ago

Fix files will be pruned to support these resolutions: C96, C192, C384, C768

Fanglin Yang will prune fix files for just those resolutions. Fix files will be frozen as of December 13th 2019 for the UFS set. I will add versioning to folders in new set and upload it to replace the current set on ftp server when ready (most likely early next week).

yangfanglin commented 4 years ago

Do you need the following fix directories ?  fix_chem   fix_gldas  fix_orog  fix_sfc_climo  fix_verif

To my opinion, you do not need them if you are not going to 1) run GFS with the chemistry-transport moudle, 2) run GLDAS module, 3) create your own new FV3 tile grid, 4) support SAR FV3 and/or generate new fixed land fields, 5) run METplux verification

arunchawla-NOAA commented 4 years ago

Fanglin

Does it make sense to keep a central place for fix files for both develop and release branches here?

Arun Chawla Chief Engineering & Implementation Branch Room 2083 National Center for Weather & Climate Prediction 5830 University Research Court College Park, MD 20740 Ph: 301-683-3740 Fx: 301-683-3703

On Fri, Dec 13, 2019 at 11:57 AM Fanglin Yang notifications@github.com wrote:

Do you need the following fix directories ? fix_chem fix_gldas fix_orog fix_sfc_climo fix_verif

To my opinion, you do not need them if you are not going to 1) run GFS with the chemistry-transport moudle, 2) run GLDAS module, 3) create your own new FV3 tile grid, 4) support SAR FV3 and/or generate new fixed land fields, 5) run METplux verification

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/15?email_source=notifications&email_token=AL5NYI2G7ESQAQH5HLOJDZ3QYO5GXA5CNFSM4JXCWJV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG2QXKY#issuecomment-565513131, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL5NYI27KX5FGEOBIDKUG7DQYO5GXANCNFSM4JXCWJVQ .

uturuncoglu commented 4 years ago

@KateFriedman-NOAA @yangfanglin

yangfanglin commented 4 years ago

Arun,

Some of the fix directories such as fix_orog and fix_sfc_climo of very large size. We kept them for development purposes.

I fUFS release is looking for a trimmed down fix directory, I do not think we can keep one development and release version.

We'd like even still keep those "old" files to support the GSM (SL and Eulerain options).

Fanglin

On Fri, Dec 13, 2019 at 12:12 PM arun chawla notifications@github.com wrote:

Fanglin

Does it make sense to keep a central place for fix files for both develop and release branches here?

Arun Chawla Chief Engineering & Implementation Branch Room 2083 National Center for Weather & Climate Prediction 5830 University Research Court College Park, MD 20740 Ph: 301-683-3740 Fx: 301-683-3703

On Fri, Dec 13, 2019 at 11:57 AM Fanglin Yang notifications@github.com wrote:

Do you need the following fix directories ? fix_chem fix_gldas fix_orog fix_sfc_climo fix_verif

To my opinion, you do not need them if you are not going to 1) run GFS with the chemistry-transport moudle, 2) run GLDAS module, 3) create your own new FV3 tile grid, 4) support SAR FV3 and/or generate new fixed land fields, 5) run METplux verification

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/ufs-community/ufs-mrweather-app/issues/15?email_source=notifications&email_token=AL5NYI2G7ESQAQH5HLOJDZ3QYO5GXA5CNFSM4JXCWJV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG2QXKY#issuecomment-565513131 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AL5NYI27KX5FGEOBIDKUG7DQYO5GXANCNFSM4JXCWJVQ

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/15?email_source=notifications&email_token=AKY5N2NFZS4ZZD2NACMJVZTQYO7BRA5CNFSM4JXCWJV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG2SNVQ#issuecomment-565520086, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKY5N2PNJYUC7EIZ3GAUXV3QYO7BRANCNFSM4JXCWJVQ .

-- Fanglin Yang, Ph.D. Physical Scientist Environmental Modeling Center National Centers for Environmental Prediction 301-6833722; fanglin.yang@noaa.gov http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/ http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/

arunchawla-NOAA commented 4 years ago

Agreed Fanglin

I am not suggesting that everything be kept all together but that we have a hierarchical directory structure for different things .

Arun Chawla Chief Engineering & Implementation Branch Room 2083 National Center for Weather & Climate Prediction 5830 University Research Court College Park, MD 20740 Ph: 301-683-3740 Fx: 301-683-3703

On Fri, Dec 13, 2019 at 12:20 PM Fanglin Yang notifications@github.com wrote:

Arun,

Some of the fix directories such as fix_orog and fix_sfc_climo of very large size. We kept them for development purposes.

I fUFS release is looking for a trimmed down fix directory, I do not think we can keep one development and release version.

We'd like even still keep those "old" files to support the GSM (SL and Eulerain options).

Fanglin

On Fri, Dec 13, 2019 at 12:12 PM arun chawla notifications@github.com wrote:

Fanglin

Does it make sense to keep a central place for fix files for both develop and release branches here?

Arun Chawla Chief Engineering & Implementation Branch Room 2083 National Center for Weather & Climate Prediction 5830 University Research Court College Park, MD 20740 Ph: 301-683-3740 Fx: 301-683-3703

On Fri, Dec 13, 2019 at 11:57 AM Fanglin Yang notifications@github.com wrote:

Do you need the following fix directories ? fix_chem fix_gldas fix_orog fix_sfc_climo fix_verif

To my opinion, you do not need them if you are not going to 1) run GFS with the chemistry-transport moudle, 2) run GLDAS module, 3) create your own new FV3 tile grid, 4) support SAR FV3 and/or generate new fixed land fields, 5) run METplux verification

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/ufs-community/ufs-mrweather-app/issues/15?email_source=notifications&email_token=AL5NYI2G7ESQAQH5HLOJDZ3QYO5GXA5CNFSM4JXCWJV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG2QXKY#issuecomment-565513131

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AL5NYI27KX5FGEOBIDKUG7DQYO5GXANCNFSM4JXCWJVQ

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/ufs-community/ufs-mrweather-app/issues/15?email_source=notifications&email_token=AKY5N2NFZS4ZZD2NACMJVZTQYO7BRA5CNFSM4JXCWJV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG2SNVQ#issuecomment-565520086 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AKY5N2PNJYUC7EIZ3GAUXV3QYO7BRANCNFSM4JXCWJVQ

.

-- Fanglin Yang, Ph.D. Physical Scientist Environmental Modeling Center National Centers for Environmental Prediction 301-6833722; fanglin.yang@noaa.gov http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/ http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/15?email_source=notifications&email_token=AL5NYI6HEHO7LEGPL6Q2OM3QYO73TA5CNFSM4JXCWJV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG2UDDI#issuecomment-565526925, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL5NYI56JW4WYDZ5NW72WT3QYO73TANCNFSM4JXCWJVQ .

rsdunlapiv commented 4 years ago

@KateFriedman-NOAA can you please provide an update on when the fixed file versioned directories will be available on the FTP site?

For everyone, we were discussing today how to handle the initial conditions that are input to chgres (that will come from NCEP http or potentially NOMADS) and how to hook that into CIME. A simple approach we are considering now is to require the user to manually populate a local directory with the initial conditions (however they are obtained) and then passing an environment variable to CIME with the name of this local directory. CIME would then set up the chgres namelist and run it on the user's downloaded data. In the future, we would like to have a way for the workflow to be able to acquire the initial condition data automatically, but the approach described here seems like a good first step. Doing this the right way will require a bit of thought about how to cache/stage data on shared platforms, and how to age files and remove them automatically.

KateFriedman-NOAA commented 4 years ago

I'm waiting to get a pruned copy of the fix files from Fanglin and then I can put the versioning on them and push them up to the ftp server.

@yangfanglin Any update on the pruned fix file set? Thanks!

yangfanglin commented 4 years ago

I have put together a trimmed down version of the GFS fix directory on WCOSS (Mars) at /gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/fv3gfs/fix_ufs. It supports C48, C96, C192, C384 and C768. The size of the directory has been reduced from the original 147GB to 7.5GB. It only contains the files that support the GFS.v15 and GFS.v16 with the FV3 dycore.

The work is done based on my best judgement. I have not tested it. Someone else needs to run a test.

KateFriedman-NOAA commented 4 years ago

Thanks @yangfanglin for making the trimmed down version.

All, I have replaced the fix directory on our ftp server with the trimmed down version:

https://ftp.emc.ncep.noaa.gov/EIB/UFS/global/fix/

fix_am.v20191213
fix_fv3_gmted2010.v20191213

The version (v20191213) corresponds to our latest set on WCOSS from which this set was made.

uturuncoglu commented 4 years ago

@KateFriedman-NOAA following file is missing under FTP, if you don't mind could you copy it.

under FIX_AM - global_co2historicaldata_glob.txt

PS: I am trying to mimic https://github.com/NOAA-EMC/global-workflow/blob/feature/gfsv16b/scripts/exglobal_fcst_nemsfv3gfs.sh under CIME.

yangfanglin commented 4 years ago

Indeed global_co2historicaldata_glob.txt is missing. Sorry for the overlook.

Kate, please copy this one over from the original fix directory.

On Wed, Dec 18, 2019 at 4:57 PM Ufuk Turunçoğlu notifications@github.com wrote:

@KateFriedman-NOAA https://github.com/KateFriedman-NOAA following file is missing under FTP

under FIX_AM - global_co2historicaldata_glob.txt

PS: I am trying to mimic https://github.com/NOAA-EMC/global-workflow/blob/feature/gfsv16b/scripts/exglobal_fcst_nemsfv3gfs.sh under CIME.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/15?email_source=notifications&email_token=AKY5N2LR2ETNKFWNMBG2I23QZKMC7A5CNFSM4JXCWJV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHHUAPY#issuecomment-567230527, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKY5N2NNGISFAF5HI4QGQGLQZKMC7ANCNFSM4JXCWJVQ .

-- Fanglin Yang, Ph.D. Physical Scientist Environmental Modeling Center National Centers for Environmental Prediction 301-6833722; fanglin.yang@noaa.gov http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/ http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/

uturuncoglu commented 4 years ago

I am not sure about following files but they might be also missing in the FTP

global/fix/fix_am.v20191213/global_soilmgldas.statsgo.t190.384.192.grb

I am trying to generate files based on the resolution given and maybe we need to use highest resolution file for it as it implemented in exglobal_fcst_nemsfv3gfs.sh

yangfanglin commented 4 years ago

Ufuk,
All the fix fields will be read in by the model and then projected onto the tiled grid. Ideally, the input data should have a much higher resolution than the T(2CASE-2) Gaussian grid resolution. We did not have all the fields at all Gaussian grid resolutions prepared either. These are the reasons why in the forecast script the fields at the T1543 resolution are used if data at the T(2CASE-2) resolution are missing.

I had all the files for t190 prepared last night. I noticed now someone changed them to symbolic links to t126 fields. Kate: did you make those changes ? Anyhow, they should be on the ftp site. I will let Kate check and answer your question.

KateFriedman-NOAA commented 4 years ago

@uturuncoglu @yangfanglin The missing file has been copied to the ftp server:

https://ftp.emc.ncep.noaa.gov/EIB/UFS/global/fix/fix_am.v20191213/global_co2historicaldata_glob.txt

KateFriedman-NOAA commented 4 years ago

I had all the files for t190 prepared last night. I noticed now someone changed them to symbolic links to t126 fields. Kate: did you make those changes ? Anyhow, they should be on the ftp site. I will let Kate check and answer your question.

I noticed a set of broken symlinks as I was going to rsync the files to the ftp server the other day and copied in the files that the symlinks were missing from the main set of fix files. Apologies if that was wrong...let me know if those symlinks should be removed and replaced with other files. I can make the edits and update the ftp set as needed.

uturuncoglu commented 4 years ago

@KateFriedman-NOAA Thanks for your help and also information.

uturuncoglu commented 4 years ago

@KateFriedman-NOAA @yangfanglin Do we need following files?

post_tag_gfs${LEVS} postxconfig-NT-GFS-TWO.txt postxconfig-NT-GFS-F00-TWO.txt params_grib2_tbl_new

They are required for inline post and also ncep_post (?) but i could not find them in the FTP. I also look at model source directory under param and some of them are in this directory but not all of them. It might be good to have them in a separate directory such as parm/post or included into the source. What do you think?

yangfanglin commented 4 years ago

They belong to the UPP package.

On Thu, Dec 19, 2019 at 12:29 PM Ufuk Turunçoğlu notifications@github.com wrote:

@KateFriedman-NOAA https://github.com/KateFriedman-NOAA @yangfanglin https://github.com/yangfanglin Do we need following files?

post_tag_gfs${LEVS} postxconfig-NT-GFS-TWO.txt postxconfig-NT-GFS-F00-TWO.txt params_grib2_tbl_new

They are required for inline post and also ncep_post (?) but i could not find them in the FTP. I also look at model source directory under param and some of them are in this directory but not all of them. It might be good to have them in a separate directory such as parm/post or included into the source. What do you think?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/15?email_source=notifications&email_token=AKY5N2KL3DGKW4VFNKPFJQDQZOVORA5CNFSM4JXCWJV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHKKV5I#issuecomment-567585525, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKY5N2KCS5LHZNSBI5FF6TDQZOVORANCNFSM4JXCWJVQ .

-- Fanglin Yang, Ph.D. Physical Scientist Environmental Modeling Center National Centers for Environmental Prediction 301-6833722; fanglin.yang@noaa.gov http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/ http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/

arunchawla-NOAA commented 4 years ago

@uturuncoglu is your testing complete ? Can this issue now be closed as the ftp site is up and running?

uturuncoglu commented 4 years ago

@KateFriedman-NOAA @yangfanglin @arunchawla-NOAA Is the FTP directory structure changed? I could not see RT directory anymore. There is a simple-test-case/ directory but it is a tar file. We were using RT directory to get some files such as tables etc.

arunchawla-NOAA commented 4 years ago

why are you using RT? That is a snapshot of one of the regression test cases. What files are you needing ? I am adding @junwang-noaa and @DusanJovic-NOAA

uturuncoglu commented 4 years ago

@arunchawla-NOAA @DusanJovic-NOAA @junwang-noaa The list of files that are retrieved from RT is

data_table diag_table field_table nems.configure -> i could create it with script for mrweather gfs_ctrl.nc sfc_data.tile.nc gfs_data.tile.nc

The nc files are required for the example test case, that will be default for the application. The table files can be retrieved from the source directory but i am not sure.

arunchawla-NOAA commented 4 years ago

the simple test case has many of the files you need

Arun Chawla Chief Engineering & Implementation Branch Room 2083 National Center for Weather & Climate Prediction 5830 University Research Court College Park, MD 20740 Ph: 301-683-3740 Fx: 301-683-3703

On Fri, Dec 20, 2019 at 3:23 PM Ufuk Turunçoğlu notifications@github.com wrote:

The list of files that are retrieved from RT is

data_table diag_table field_table nems.configure -> i could create it with script for mrweather gfs_ctrl.nc sfc_data.tile.nc gfs_data.tile.nc

The nc files are required for the example test case, that will be default for the application. The table files can be retrieved from the source directory but i am not sure.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/15?email_source=notifications&email_token=AL5NYI3E4IDRLBR7GJLAFB3QZUSUFA5CNFSM4JXCWJV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHOCLIA#issuecomment-568075680, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL5NYI54X6DICECC7TEVXXTQZUSUFANCNFSM4JXCWJVQ .

uturuncoglu commented 4 years ago

I think that it would be nice to have sample set of ICs that could be used to run the model without chgres. I also did not implement restart capability yet. We might need some files for it. BTW, is there any documentation that has information about restarting the standalone FV3.

uturuncoglu commented 4 years ago

@arunchawla-NOAA Then, we need to extract it just as we did before for fv3_gfdlmprad.

uturuncoglu commented 4 years ago

@arunchawla-NOAA @GeorgeGayno-NOAA @KateFriedman-NOAA @yangfanglin How do we handle required fixed resolution dependent input files for CHGRES? So, following files needed for each supported resolution and need to be placed under FTP

C96.facsf.*.nc
C96.maximum_snow_albedo.*.nc
C96.slope_type.*.nc
C96.snowfree_albedo.*.nc
C96.soil_type.*.nc
C96.substrate_temperature.*.nc
C96.vegetation_greenness.*.nc
C96.vegetation_type.*.nc