search

ufs-community / ufs-mrweather-app

UFS Medium-Range Weather Application
Other
23 stars 23 forks source link

chgres update #154

Closed arunchawla-NOAA closed 4 years ago

arunchawla-NOAA commented 4 years ago

chgres update for

---providing a fix for extreme ocean temperatures

---new input files from gfs that are coming with the v 16 implementation (exact date these files will be available in operations is still not clear)

GeorgeGayno-NOAA commented 4 years ago

Both updates were tested and merged to 'develop'.

For details about the ocean temperature fix, see: https://github.com/NOAA-EMC/UFS_UTILS/issues/59

For details on the capability to ingest GFS v16 gaussian netcdf files (which replace the v15 nemsio files), see: https://github.com/NOAA-EMC/UFS_UTILS/issues/38

ligiabernardet commented 4 years ago

Thanks, @GeorgeGayno-NOAA for these upgrades. Would be able to update the documentation for the release at https://github.com/NOAA-EMC/UFS_UTILS/tree/release/public-v1?

GeorgeGayno-NOAA commented 4 years ago

Thanks, @GeorgeGayno-NOAA for these upgrades. Would be able to update the documentation for the release at https://github.com/NOAA-EMC/UFS_UTILS/tree/release/public-v1?

Yes, I can update the documentation. However, I am not familiar with the "rst" format.

ligiabernardet commented 4 years ago

Mostly you should be able to update the rst documentation following what is there already. Julie Schramm at DTC is an expert on Sphinx that can support you. The Sphinx page is at https://www.sphinx-doc.org/. A very basic example of how to compile the documentation and generate the HTML can be seen in the README at https://github.com/NCAR/ccpp-doc. Once you have updates, we can also see how they are turning out in ReadTheDocs. Let us know if we can help.

ligiabernardet commented 4 years ago

@GeorgeGayno-NOAA When do you anticipate putting the upgrades in the release/public-v1 branch so we can test for the release? Also, How is it going with the documentation? Let us know if you need any help.

GeorgeGayno-NOAA commented 4 years ago

@GeorgeGayno-NOAA When do you anticipate putting the upgrades in the release/public-v1 branch so we can test for the release? Also, How is it going with the documentation? Let us know if you need any help.

Has anyone created a branch for this update?

climbfuji commented 4 years ago

@GeorgeGayno-NOAA When do you anticipate putting the upgrades in the release/public-v1 branch so we can test for the release? Also, How is it going with the documentation? Let us know if you need any help.

Has anyone created a branch for this update?

Updates for release 1.1 will be made to the existing release/public-v1 branches.

GeorgeGayno-NOAA commented 4 years ago

@ligiabernardet I updated the chgres_cube code and documentation in the release branch in my fork.

I did not try running the chgres_cube program. I can't figure out how to compile the release branch on Hera.

I don't know the details of where the GFS v16 NetCDF history files will reside on NOMADS. So, I left that section of the documentation blank for now.

ligiabernardet commented 4 years ago

@GeorgeGayno-NOAA I was able to run the UFS MRW v1.0 on Hera today by following the instructions provided by Linlin in this forum post https://forums.ufscommunity.org/threads/module-files-loading-problem-hera-and-jet. It involves a change to file modulefiles/hera.intel/fv3.

Would you be able to provide us with a sample of NetCDF file(s) on Hera?

GeorgeGayno-NOAA commented 4 years ago

@GeorgeGayno-NOAA I was able to run the UFS MRW v1.0 on Hera today by following the instructions provided by Linlin in this forum post https://forums.ufscommunity.org/threads/module-files-loading-problem-hera-and-jet. It involves a change to file modulefiles/hera.intel/fv3.

Would you be able to provide us with a sample of NetCDF file(s) on Hera?

Some sample GFS v16 NetCDF data is here: /scratch1/NCEPDEV/da/George.Gayno/noscrub/chgres.gfsv16/gfs.20200202/00

A sample chgres_cube namelist to convert this v16 data is here: /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/gfsv16/config.ligia.nml

Note: "input_type" must be set to "gaussian_netcdf"

ligiabernardet commented 4 years ago

Thanks, George. @rsdunlapiv Perhaps you can get this sample to @uturuncoglu and @jedwards4b so they can try it with the App?

ligiabernardet commented 4 years ago

@GeorgeGayno-NOAA Have you been able to get the updated chgres_cube into the release branch (existing release/public-v1 branch)?

GeorgeGayno-NOAA commented 4 years ago

@GeorgeGayno-NOAA Have you been able to get the updated chgres_cube into the release branch (existing release/public-v1 branch)?

The updates are still in my fork (https://github.com/GeorgeGayno-NOAA/UFS_UTILS/tree/release/public-v1). I am hoping others could run a quick test to verify it is working. I can assist with any testing.

climbfuji commented 4 years ago

@pjpegion you were in charge of the testing team for the UFS MRW App v1.0, I am in charge for the UFS SRW App v1.0. How can we make sure that these changes get tested (together with updates to the build system - NCEPLIBS-external) in the next week so that we can roll out this release by mid-to-end of August?

@arunchawla-NOAA @lbnance @mvertens

pjpegion commented 4 years ago

@climbfuji I'm not sure how to do this. Are there regression tests for the mr-weather-app like there is for the model?

climbfuji commented 4 years ago

@pjpegion @jedwards4b @uturuncoglu Yes, I think there are tests for the entire app - our NCAR CIME colleagues ran those prior to the release in March.

climbfuji commented 4 years ago

@GeorgeGayno-NOAA I pulled your release/public-v1 branch into the NOAA-EMC NCEPLIBS release/public-v1 for testing (not committed, of course). If this builds on my mac, I will build it on hera so that @ligiabernardet can test it with the netCDF input data.

climbfuji commented 4 years ago

@GeorgeGayno-NOAA the compilation of this UFS_UTILS fails on hera with Intel-18.0.4:

[ 99%] Building Fortran object UFS_UTILS/sorc/chgres_cube.fd/CMakeFiles/chgres_cube.exe.dir/atmosphere.F90.o
cd /scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/UFS_UTILS/sorc/chgres_cube.fd && /apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/bin/intel64/ifort  -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/NCEPLIBS-nemsio/include -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/NCEPLIBS-sfcio/include_4 -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/NCEPLIBS-sigio/include_4 -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/NCEPLIBS-bacio/include_4 -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/NCEPLIBS-sp/include_d -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/NCEPLIBS-w3nco/include_d -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/mod -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/include -I/apps/netcdf/4.7.0/intel/18.0.5.274/include -I/apps/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/include  -g -traceback -O3   -c /scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/UFS_UTILS/sorc/chgres_cube.fd/atmosphere.F90 -o CMakeFiles/chgres_cube.exe.dir/atmosphere.F90.o
[ 99%] Building Fortran object UFS_UTILS/sorc/chgres_cube.fd/CMakeFiles/chgres_cube.exe.dir/surface.F90.o
cd /scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/UFS_UTILS/sorc/chgres_cube.fd && /apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/bin/intel64/ifort  -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/NCEPLIBS-nemsio/include -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/NCEPLIBS-sfcio/include_4 -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/NCEPLIBS-sigio/include_4 -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/NCEPLIBS-bacio/include_4 -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/NCEPLIBS-sp/include_d -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build/NCEPLIBS-w3nco/include_d -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/mod -I/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/include -I/apps/netcdf/4.7.0/intel/18.0.5.274/include -I/apps/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/include  -g -traceback -O3   -c /scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/UFS_UTILS/sorc/chgres_cube.fd/surface.F90 -o CMakeFiles/chgres_cube.exe.dir/surface.F90.o
/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/UFS_UTILS/sorc/chgres_cube.fd/surface.F90(2348): warning #7319: This argument's data type is incompatible with this intrinsic procedure; procedure assumed EXTERNAL.   [ALOG]
     DF = ALOG(( PSIS*GRAV/HLICE ) * ( ( 1.+CK*SWL )**2. ) *      &
-----------------------------------------------------------^
/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/UFS_UTILS/sorc/chgres_cube.fd/surface.F90(2348): error #6404: This name does not have a type, and must have an explicit type.   [ALOG]
     DF = ALOG(( PSIS*GRAV/HLICE ) * ( ( 1.+CK*SWL )**2. ) *      &
----------^
compilation aborted for /scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/UFS_UTILS/sorc/chgres_cube.fd/surface.F90 (code 1)
make[2]: *** [UFS_UTILS/sorc/chgres_cube.fd/CMakeFiles/chgres_cube.exe.dir/surface.F90.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build'
make[1]: *** [UFS_UTILS/sorc/chgres_cube.fd/CMakeFiles/chgres_cube.exe.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build'
make[1]: *** [NCEPLIBS-post/sorc/ncep_post.fd/CMakeFiles/nceppost.dir/all] Error 2
make[1]: Leaving directory `/scratch1/BMC/gmtb/software/NCEPLIBS-ufs-v1.1.0.beta01/intel-18.0.5.274/impi-2018.0.4/src/NCEPLIBS/build'
make: *** [all] Error 2

On macOS with clang-9.0.0+gfortran-9.2.0 / gcc-9.2.0+gfortran -9.2.0:

[ 89%] Building Fortran object UFS_UTILS/sorc/chgres_cube.fd/CMakeFiles/chgres_cube.exe.dir/static_data.F90.o
[ 89%] Building Fortran object UFS_UTILS/sorc/chgres_cube.fd/CMakeFiles/chgres_cube.exe.dir/surface.F90.o
/usr/local/gnu_mpionly/NCEPLIBS-ufs-v1.1.0/src/NCEPLIBS/UFS_UTILS/sorc/chgres_cube.fd/surface.F90:2349:41:

 2349 |         ( SMCMAX/(SMC-SWL) )**BX) - ALOG(-(TKELV-frz_h2o)/TKELV)
      |                                         1
Error: Type of argument 'x' in call to 'alog' at (1) should be REAL(4), not REAL(8)
make[2]: *** [UFS_UTILS/sorc/chgres_cube.fd/CMakeFiles/chgres_cube.exe.dir/surface.F90.o] Error 1
make[1]: *** [UFS_UTILS/sorc/chgres_cube.fd/CMakeFiles/chgres_cube.exe.dir/all] Error 2
make: *** [all] Error 2

When you make those changes in your UFS_UTILS branch, can you also please update the CMakeModules submodule to point to the following hash?

commit 29e34bc2afc40890dc8532b85013de12432d68c5 (HEAD, tag: v1.1.0, origin/develop, origin/HEAD, develop)
Author: Dom Heinzeller <dom.heinzeller@icloud.com>
Date:   Thu Jul 30 19:40:44 2020 -0600

    Remove unused cmake macros (#40)

Commands:

cd cmake
git remote update
git fetch --tags
git checkout v1.1.0
cd ..
git add cmake
git commit -m "Update submodule pointer for CMakeModules"
GeorgeGayno-NOAA commented 4 years ago

@climbfuji Try to build chgres_cube now.

climbfuji commented 4 years ago

@GeorgeGayno-NOAA thanks, I am getting past this error, but now the linker step fails (wgrib2api). I'll work on this tomorrow Monday.

GeorgeGayno-NOAA commented 4 years ago

@GeorgeGayno-NOAA thanks, I am getting past this error, but now the linker step fails (wgrib2api). I'll work on this tomorrow Monday.

Sorry. I am unable to figure out how to compile the release branch on Hera. So I am flying somewhat blind.

arunchawla-NOAA commented 4 years ago

@GeorgeGayno-NOAA and @climbfuji were you able to get this to build on Mac/Hera? If not then I will see if we can get additional help from Kyle here.

climbfuji commented 4 years ago

@GeorgeGayno-NOAA and @climbfuji were you able to get this to build on Mac/Hera? If not then I will see if we can get additional help from Kyle here.

It's a simple linker flag that is missing ... will have it done in a bit.

climbfuji commented 4 years ago

I have a bunch of PRs out, including one for NOAA-EMC UFS_UTILS, branch release/public-v1. This PR contains all of @GeorgeGayno-NOAA's changes (with his original ownerships on those commits, of course) and a few small updates for the build system. With those, I can build NCEPLIBS-external with the new NCEPLIBS-wgrib2, and afterwards the entire NCEPLIBS release/public-v1 on hera.intel, cheyenne.intel, cheyenne.gnu, macosx.gccgfortran and macosx.clanggfortran. More tests to follow, in particular "nasty" systems such as the Cray system gaea.intel.

https://github.com/NOAA-EMC/NCEPLIBS-grib_util/pull/22 https://github.com/NOAA-EMC/NCEPLIBS-wgrib2/pull/12 (still a draft PR because of an open question) https://github.com/NOAA-EMC/EMC_post/pull/160 https://github.com/NOAA-EMC/NCEPLIBS-prod_util/pull/18 https://github.com/NOAA-EMC/UFS_UTILS/pull/130

Once these are all merged, I will create the umbrella update PRs for NCEPLIBS-external and NCEPLIBS release/public-v1.

climbfuji commented 4 years ago

Most of the PRs have been merged, remaining one is the EMC_post PR. Once this is in, I will create the follow-up PRs for NCEPLIBS-external and NCEPLIBS umbrella. A beta01 version of the libraries is installed on hera so that somebody can test the netCDF ingest.

ligiabernardet commented 4 years ago

@climbfuji Where do I find the NCEPLIBS-external and NCEPLIBS on Hera?

climbfuji commented 4 years ago

@climbfuji Where do I find the NCEPLIBS-external and NCEPLIBS on Hera?

Please see the note I posted in the build system improvement issue https://github.com/ufs-community/ufs-mrweather-app/issues/157: https://github.com/ufs-community/ufs-mrweather-app/issues/157#issuecomment-671692724

ligiabernardet commented 4 years ago

@GeorgeGayno-NOAA I was able to run the C96 chgres_cube standalone on Hera using GRIB2 and netCDF for 02/02/2020. The results of the run using netCDF look earth-like in the troposphere. I noticed substantial differences between the two results in the highest model levels (levels 0, 1, 2). For example T differences are greater than 40 K in certain places at Level 0. Is that because the GRIB2 files do not have enough information up there? Also, thank you for the documentation updates.

GeorgeGayno-NOAA commented 4 years ago

@GeorgeGayno-NOAA I was able to run the C96 chgres_cube standalone on Hera using GRIB2 and netCDF for 02/02/2020. The results of the run using netCDF look earth-like in the troposphere. I noticed substantial differences between the two results in the highest model levels (levels 0, 1, 2). For example T differences are greater than 40 K in certain places at Level 0. Is that because the GRIB2 files do not have enough information up there? Also, thank you for the documentation updates.

@ligiabernardet I agree with your assessment. The GRIB2 data has information up to 1 mb. The 127 level netCDF files have information up to about 1 Pa. So depending on your model top, I would expect differences. chgres_cube is not very sophisticated. It will simply persist the 1 mb values upward to the model top.