CIME update for chgres_cube "gaussian_netcdf" namelist option

[arunchawla-NOAA]

CIME needs to be updated to account for a new namelist option for chgres_cube. With the option of using netcdf files the input_type should now have a third option "gaussian_netcdf"

[ligiabernardet]

CIME must also recognize the netCDF file names, which are of the form gfs.tHHz.atmf000.nc gfs.tHHz.sfcf000.nc

Examples on Hera at /scratch1/NCEPDEV/da/George.Gayno/noscrub/chgres.gfsv16/gfs.20200202/00

[arunchawla-NOAA]

Also we need to update FV3 namelist options for damping coefficients. These were erroneously set for 127 level configuration. Need to be for 64 levels. See this thread for details

https://forums.ufscommunity.org/threads/strength-second-order-damping-top-layers

[uturuncoglu]

@arunchawla-NOAA sure. @rsdunlapiv could you move those files to Cheyenne or Orion? Thanks.

[rsdunlapiv]

@uturuncoglu Yes I can move the files. I have to have my hera certificate resigned, so it said it could be a day or two. Will try again tomorrow.

[climbfuji]

@uturuncoglu Yes I can move the files. I have to have my hera certificate resigned, so it said it could be a day or two. Will try again tomorrow.

Usually the certificate only takes an hour or so, even though they say 1-2 days.

[rsdunlapiv]

@climbfuji Execellent, I will try again shortly.

[rsdunlapiv]

@uturuncoglu I moved to files to Cheyenne: /glade/scratch/dunlap/ufs_xfer

[uturuncoglu]

Okay. Thanks @rsdunlapiv.

[uturuncoglu]

@climbfuji @ligiabernardet i think we need to update NCEPLIBS to handle netCDF files (especially CHGRES). If you don't mind could you point me the instructions as well as hashes that needs to be used to test the netCDF input? If you all provide a example CHGRES script that would be great. BTW, is there any change in the post-processing side?

[climbfuji]

I have NCEPLIBS ufs-v1.1.0 installed on almost all the platforms by now. For testing those with CIME, can you use hera or cheyenne? The new modulefiles are

# cheyenne.gnu
module use -a /glade/p/ral/jntp/GMTB/tools/modulefiles/gnu-8.3.0/mpt-2.19
module load NCEPlibs/1.1.0

# cheyenne.intel
module use -a /glade/p/ral/jntp/GMTB/tools/modulefiles/intel-19.0.5/mpt-2.19
module load NCEPlibs/1.1.0

# hera.intel
module use -a /scratch1/BMC/gmtb/software/modulefiles/intel-18.0.5.274/impi-2018.0.4
module load  NCEPlibs/1.1.0

I believe none of the prerequisite modules (compiler, ...) has changed for those, but I am not sure. For other systems such as gaea, the definitely changed. Can you please make sure for every system that you are testing/configuring for MRW v1.1 that the modules loaded in the CIME config match those in the modulefiles for the preconfigured platforms?

For the supported platforms, we need to check against the updated doc/README_*.txt files in the NCEPLIBS-external repo (but I am still working on the last documentation updates, hence this needs to wait for a bit).

If CIME aborts in case a module load fails due to wrong prerequisites, we should be on the safe side.

Thanks!

[uturuncoglu]

@climbfuji That is really great! Thanks. I could use cheyenne.intel for the tests. I'll keep you posted about the progress.

[ligiabernardet]

@Ufuk Turuncoglu turuncu@ucar.edu I was able to run the C96 chgres_cube standalone with the netCDF input on Hera. I have placed the script along with two namelists (one for GRIB2 input and one for netCDF input) on Cheyenne for you at /glade/u/home/bernarde/chgres_test. The updated chgres_cube documentation, describing the namelist entries for netCDF output, is at https://ufs-utils.readthedocs.io/en/latest/chgres_cube.html#chgres-cube-namelist-options

On Wed, Aug 12, 2020 at 11:30 AM Ufuk Turunçoğlu notifications@github.com wrote:

@climbfuji https://github.com/climbfuji That is really great! Thanks. I could use cheyenne.intel for the tests. I'll keep you posted about the progress.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/162#issuecomment-673010073, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7WQAWJHZU5ENAHFG4G7LLSALGSZANCNFSM4P2CULRA .

[rsdunlapiv]

@uturuncoglu this is one of the main items that we need to take care of before the release. Please keep us posted as you try the modules provided by @climbfuji.

[uturuncoglu]

@ligiabernardet that is great! I'll look at them. Currently, I have problem related with the unification of UFSATM CIME interface and updating the model. The updated model can not be built with existing interface (more information in https://github.com/ufs-community/ufs-mrweather-app/issues/169) and needs to be handled first. Then, I could start to the initial implementation of supporting new data format.

@rsdunlapiv Sure, i'll keep you posted about the progress.

[uturuncoglu]

@ligiabernardet what is the last three digit in the file - gfs.tHHz.atmf000.nc. I think it is forecast hour of input data but I am asking to be sure. Do we need only support 000? i.e. analysis. Is there any use case that user might need to start some other time, for example gfs.tHHz.atmf003.nc

[uturuncoglu]

@ligiabernardet BTW, tHHz part could change also?

[ligiabernardet]

tHH: This is the time of GFS initialization, and could change to be 00, 06, 12, 18. No other options, as the model only starts 4 times a day. f000: 000 is the forecast time, that is, 000 is the initial time. In my opinion, all we need is 000 (not 003 etc.)

On Wed, Aug 19, 2020 at 12:32 PM Ufuk Turunçoğlu notifications@github.com wrote:

@ligiabernardet https://github.com/ligiabernardet BTW, tHHz part could change also?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/162#issuecomment-676590440, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7WQAXWV2STZGA6S646XDLSBQLDJANCNFSM4P2CULRA .

[uturuncoglu]

@ligiabernardet Thanks for the information. That is really helpful and simply the CIME interface. I have already implemented gaussian_netcdf option and I am testing it now. I'll let you know when I have successful run with NetCDF files.

[uturuncoglu]

@ligiabernardet @climbfuji I run the CHGRES with netCDF input and it is failing without giving any error messages. I am using C96 resolution to test and grib2 works without any problem in same number of processor. I just wonder do we need to allocate more processor, if we are processing netCDF input in CHGRES? Here is my config.nml

&config
  atm_files_input_grid = "gfs.t00z.atmf000.nc"
  convert_atm = .true.
  convert_nst = .true.
  convert_sfc = .true.
  cycle_day = 2
  cycle_hour = 0
  cycle_mon = 2
  data_dir_input_grid = "/glade/p/cesmdata/cseg/ufs_inputdata/icfiles/202002/20200202"
  fix_dir_target_grid = "INPUT"
  input_type = "gaussian_netcdf"
  mosaic_file_target_grid = "INPUT/C96_mosaic.nc"
  orog_dir_target_grid = "INPUT"
  orog_files_target_grid = "oro_data.tile1.nc", "oro_data.tile2.nc",
      "oro_data.tile3.nc", "oro_data.tile4.nc", "oro_data.tile5.nc",
      "oro_data.tile6.nc"
  sfc_files_input_grid = "gfs.t00z.sfcf000.nc"
  tracers = "sphum", "liq_wat", "o3mr", "ice_wat", "rainwat", "snowwat",
      "graupel"
  tracers_input = "spfh", "clwmr", "o3mr", "icmr", "rwmr", "snmr", "grle"
  vcoord_file_target_grid = "INPUT/global_hyblev.l65.txt"
/

You could reach full CHGRES log from here,

/glade/scratch/turuncu/ufs-mrweather-app-workflow.c96v2/run/chgres_cube.200819-144825.log

In this case, I am using NCEPLIBS from /glade/p/ral/jntp/GMTB/tools/modulefiles/gnu-8.3.0/mpt-2.19, NCEPlibs/1.1.0. Is there any change in global_hyblev.l65.txt etc. after previous release. We are using global_hyblev.* files from https://ftp.emc.ncep.noaa.gov/EIB/UFS/global/fix/fix_am.v20191213/.

[ligiabernardet]

Ufuk, To my knowledge there is no change in global_hyblev.l65.txt etc. Regarding the number of processors, the documentation https://ufs-utils.readthedocs.io/en/latest/chgres_cube.html#chgres-cube-namelist-options says it should be a multiple of 6. I used 6 in my test for C96. My run of the C96 chgres_cube standalone with the netCDF input on Hera. I have placed the script along with two namelists (one for GRIB2 input and one for netCDF input) on Cheyenne for you at /glade/u/home/bernarde/chgres_test. By comparing our namelists, I see that you do not have variable varmap in your namelist. According to the documentation https://ufs-utils.readthedocs.io/en/latest/chgres_cube.html#chgres-cube-namelist-options, this variable should be in the namelist. Could this be the problem?

On Wed, Aug 19, 2020 at 3:03 PM Ufuk Turunçoğlu notifications@github.com wrote:

@ligiabernardet https://github.com/ligiabernardet @climbfuji https://github.com/climbfuji I run the CHGRES with netCDF input and it is failing without giving any error messages. I am using C96 resolution to test and grib2 works without any problem in same number of processor. I just wonder do we need to allocate more processor, if we are processing netCDF input in CHGRES? Here is my config.nml

&config atm_files_input_grid = "gfs.t00z.atmf000.nc" convert_atm = .true. convert_nst = .true. convert_sfc = .true. cycle_day = 2 cycle_hour = 0 cycle_mon = 2 data_dir_input_grid = "/glade/p/cesmdata/cseg/ufs_inputdata/icfiles/202002/20200202" fix_dir_target_grid = "INPUT" input_type = "gaussian_netcdf" mosaic_file_target_grid = "INPUT/C96_mosaic.nc" orog_dir_target_grid = "INPUT" orog_files_target_grid = "oro_data.tile1.nc", "oro_data.tile2.nc", "oro_data.tile3.nc", "oro_data.tile4.nc", "oro_data.tile5.nc", "oro_data.tile6.nc" sfc_files_input_grid = "gfs.t00z.sfcf000.nc" tracers = "sphum", "liq_wat", "o3mr", "ice_wat", "rainwat", "snowwat", "graupel" tracers_input = "spfh", "clwmr", "o3mr", "icmr", "rwmr", "snmr", "grle" vcoord_file_target_grid = "INPUT/global_hyblev.l65.txt" /

You could reach full CHGRES log from here,

/glade/scratch/turuncu/ufs-mrweather-app-workflow.c96v2/run/chgres_cube.200819-144825.log

In this case, I am using NCEPLIBS from /glade/p/ral/jntp/GMTB/tools/modulefiles/gnu-8.3.0/mpt-2.19, NCEPlibs/ 1.1.0. Is there any change in global_hyblev.l65.txt etc. after previous release. We are using global_hyblev.* files from https://ftp.emc.ncep.noaa.gov/EIB/UFS/global/fix/fix_am.v20191213/.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/162#issuecomment-676699831, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7WQARRXPTNCSWFVWGRO53SBQ4Z5ANCNFSM4P2CULRA .

[uturuncoglu]

@ligiabernardet Thanks for the information. I doubled the number of processor from 36 to 72 for C96 and it seems it process the data and produce the inputs. I did not test the model yet with this input but it seems that "gaussian_netcdf" requires more resource in terms of memory. It would be nice to document it in somewhere. Maybe it could be optimized in CHGRES side. I am not sure.

[uturuncoglu]

@ligiabernardet varmap is not a requirement for netCDF if I look at the link. It is just for GRIB2.

[ligiabernardet]

Great to know. BTW, I looked again at the documentation and you should not need varmap for netCDF.

On Wed, Aug 19, 2020 at 3:21 PM Ufuk Turunçoğlu notifications@github.com wrote:

@ligiabernardet https://github.com/ligiabernardet Thanks for the information. I doubled the number of processor from 36 to 72 for C96 and it seems it process the data and produce the inputs. I did not test the model yet with this input but it seems that "gaussian_netcdf" requires more resource in terms of memory. It would be nice to document it in somewhere. Maybe it could be optimized in CHGRES side. I am not sure.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/162#issuecomment-676716382, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7WQAX4D3X6V7MA55P7QCTSBQ65FANCNFSM4P2CULRA .

[uturuncoglu]

@ligiabernardet @arunchawla-NOAA @rsdunlapiv @climbfuji i have successful run with netCDF input. I also updated app. So, if you want to test it here is the instructions:

git clone https://github.com/ufs-community/ufs-mrweather-app.git
cd ufs-mrweather-app
git checkout ufs-release-v1.1
./manage_externals/checkout_externals
cd cime/scripts
UFS_DRIVER=nems CIME_MODEL=ufs ./create_newcase --compset GFSv15p2 --res C96 --case ufs-mrweather-app-workflow.c96 --workflow ufs-mrweather
cd ufs-mrweather-app-workflow.c96
./case.setup
NOTE:  add input_type = "gaussian_netcdf" to user_nl_ufsatm
./xmlchange RUN_STARTDATE=2020-02-02
./xmlchange DOUT_S=FALSE
./xmlchange STOP_OPTION=nhours
./xmlchange STOP_N=36
./xmlchange JOB_WALLCLOCK_TIME=00:30:00
./xmlchange USER_REQUESTED_WALLTIME=00:30:00
./case.build
NOTE: you might need to issue ./case.build command again if you get error from radiation_aerosols.f, in the second one the model builds fine. I am not sure what is wrong with this file at this moment.
NOTE: you need to download files manually and place in your initial condition directory.
./case.submit 
NOTE: if CHGRES fails without any particular error. Just try to increase the number of core used by CHGRES using
./xmlchange task_count=144 --subgroup case.chgres and maybe more (for C96 i set 144 as default if you select gaussian_netcdf).

BTW, if you want to test gfs3 and gfs4 input, you just need to add resolution_for_grib2 = "1.0" or resolution_for_grib2 = "0.5" to user_nl_ufsatm. If you don't add anything gfs4 is the default unless if you have netcdf file in same directory. Then, you need to set input_type explicitly. The CIME interface also handles automatic retrieval of GRIB2 files.

@climbfuji We need to update the supported machines to use NCEPLIBS 1.1.0 also.

[ligiabernardet]

@Ufuk Turuncoglu - NOAA Affiliate ufuk.turuncoglu@noaa.gov I believe this commit went to the wrong branch. We are using release/public-v1. This is the naming convention that was agreed upon. It is a copy of the branch that was used for the last release. We are simply adding to the same branch that was used for the last release. Could you please direct your commit to that branch? Sorry for the confusion.

Is the App ready to run on Hera? In other words, have the modules etc. already been updated? Just asking because I am more familiar with Hera.

On Thu, Aug 20, 2020 at 9:58 AM Ufuk Turunçoğlu notifications@github.com wrote:

@ligiabernardet https://github.com/ligiabernardet @arunchawla-NOAA https://github.com/arunchawla-NOAA @rsdunlapiv https://github.com/rsdunlapiv @climbfuji https://github.com/climbfuji i have successful run with netCDF input. I also updated app. So, if you want to test it here is the instructions:

git clone https://github.com/ufs-community/ufs-mrweather-app.git cd ufs-mrweather-app git checkout ufs-release-v1.1 ./manage_externals/checkout_externals cd cime/scripts UFS_DRIVER=nems CIME_MODEL=ufs ./create_newcase --compset GFSv15p2 --res C96 --case ufs-mrweather-app-workflow.c96 --workflow ufs-mrweather cd ufs-mrweather-app-workflow.c96 ./case.setup NOTE: add input_type = "gaussian_netcdf" to user_nl_ufsatm ./xmlchange RUN_STARTDATE=2020-02-02 ./xmlchange DOUT_S=FALSE ./xmlchange STOP_OPTION=nhours ./xmlchange STOP_N=36 ./xmlchange JOB_WALLCLOCK_TIME=00:30:00 ./xmlchange USER_REQUESTED_WALLTIME=00:30:00 ./case.build NOTE: you might need to issue ./case.build command again if you get error from radiation_aerosols.f, in the second one the model builds fine. I am not sure what is wrong with this file at this moment. NOTE: you need to download files manually and place in your initial condition directory. ./case.submit NOTE: if CHGRES fails without any particular error. Just try to increase the number of core used by CHGRES using ./xmlchange task_count=144 --subgroup case.chgres and maybe more (for C96 i set 144 as default if you select gaussian_netcdf).

BTW, if you want to test gfs3 and gfs4 input, you just need to add resolution_for_grib2 = "1.0" or resolution_for_grib2 = "0.5" to user_nl_ufsatm. If you don't add anything gfs4 is the default unless if you have netcdf file in same directory. Then, you need to set input_type explicitly. The CIME interface also handles automatic retrieval of GRIB2 files.

@climbfuji https://github.com/climbfuji We need to update the supported machines to use NCEPLIBS 1.1.0 also.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/162#issuecomment-677752026, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7WQAVC6YKKIXHDF35FVM3SBVBZDANCNFSM4P2CULRA .

[uturuncoglu]

@ligiabernardet i am not sure. I did it intentionally becuase we were using ufs-release-v1.0 branch for development and then it became publicly available as release/public-v1. I did not want to push modifications that are not fully tested on all supported platforms to release/public-v1. We still need to run full test suite on Cheyenne, Stampede, Orion etc. Anyway, if you want me to update release/public-v1 branch that is fine but then the external users will get the updated code. Maybe I am missing something in here. If so, please clarify it.

[uturuncoglu]

@ligiabernardet i have no access to Hera and i don't know it works or not in there. We also need to update NCEPLIBS module for all platforms that are supported in release 1.0. So, my understanding is that we are not ready yet to go public.

[ligiabernardet]

Ufuk, okay, we can use the branch you proposed (ufs-release-v1.1) for the test. Once we have something we are more confident on, it should go to release/public-v1. Dom has already rolled out the NCEPLIBS module in all supported platforms. Please see https://github.com/ufs-community/ufs-mrweather-app/issues/157. Is anything missing?

On Thu, Aug 20, 2020 at 10:22 PM Ufuk Turunçoğlu notifications@github.com wrote:

@ligiabernardet https://github.com/ligiabernardet i have no access to Hera and i don't know it works or not in there. We also need to update NCEPLIBS module for all platforms that are supported in release 1.0. So, my understanding is that we are not ready yet to go public.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/162#issuecomment-678029943, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7WQASXPKW7JWDNWE5XUJ3SBXZALANCNFSM4P2CULRA .

[uturuncoglu]

@ligiabernardet That is great. I'll check the platforms and make modifications in the CIME side accordingly. I could test the model on Cheyenne, Stampede and maybe Orion but other platforms need to be tested also.

[ligiabernardet]

@Ufuk Turuncoglu turuncu@ucar.edu Would it be possible for you to update config_machines.xml based on https://github.com/ufs-community/ufs-weather-model/blob/release/public-v1/modulefiles for Hera/Intel? That way I can also test on Hera.

On Thu, Aug 20, 2020 at 10:31 PM Ufuk Turunçoğlu notifications@github.com wrote:

@ligiabernardet https://github.com/ligiabernardet That is great. I'll check the platforms and make modifications in the CIME side accordingly. I could test the model on Cheyenne, Stampede and maybe Orion but other platforms need to be tested also.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/162#issuecomment-678032195, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7WQAUFMH3GUDP2FQ2HMO3SBX2DHANCNFSM4P2CULRA .

[uturuncoglu]

@ligiabernardet sure. I'll do it tomorrow first thing and let you know.

[ligiabernardet]

CIME can now process the netCDF data. Minimally tested so far, but it is working.