search

ufs-community / ufs-mrweather-app

UFS Medium-Range Weather Application
Other
23 stars 23 forks source link

Implement model regression tests with CIME #43

Closed rsdunlapiv closed 4 years ago

rsdunlapiv commented 4 years ago

A basic "smoke test" has been implemented with CIME - this is a kind of sanity check to ensure that the model builds and runs for a short time. We need to identify what additional regression tests would be most beneficial for the release and implement them.

GeorgeGayno-NOAA commented 4 years ago

@junwang-noaa and @GeorgeGayno-NOAA I am trying to complete to port of CIME to Hera. Can you please let me know where the ECM_post executable and chgres_cube executable are located? Are these up to date with the release branch versions?

I don't know if the release branch will compile on Hera. I have not tried. chgres_cube in 'develop' will work with nemsio and grib2. I can compile 'develop' on Hera.

rsdunlapiv commented 4 years ago

@GeorgeGayno-NOAA It seems important to have the release versions of these tools on Hera so we can test there. Would you mind giving it a shot?

arunchawla-NOAA commented 4 years ago

@kgerheiser can you build the libraries and externals tool set on HERA that Rocky can use?

climbfuji commented 4 years ago

PLEASE. We are working on this as fast as we can, and I have been in contact with Rocky, Mark and Kyle that I am going to do this as soon as the last issues of the cmake build are fixedd.

On Jan 29, 2020, at 10:44 AM, arun chawla notifications@github.com wrote:

@kgerheiser https://github.com/kgerheiser can you build the libraries and externals tool set on HERA that Rocky can use?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/43?email_source=notifications&email_token=AB5C2RLGIRGYRE7C4XK3PJ3RAG57BA5CNFSM4J6DO3V2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKIDIVI#issuecomment-579875925, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2RL45ZDYG6A4M6U7GZ3RAG57BANCNFSM4J6DO3VQ.

kgerheiser commented 4 years ago

Fix incoming. Hopefully it will build now.

kgerheiser commented 4 years ago

Fixed and I was able to build NCEPLIBS on Hera

climbfuji commented 4 years ago

Great! I am building and installing it in an "officially supported" location for the CIME folks to test. Thanks for making this work so quickly.

On Jan 29, 2020, at 1:07 PM, Kyle Gerheiser notifications@github.com wrote:

Fixed and I was able to build on Hera

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/43?email_source=notifications&email_token=AB5C2ROM7UHEEDAQQJSP36TRAHOYHA5CNFSM4J6DO3V2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKISBOQ#issuecomment-579936442, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2RLKZVQZO7UX72BITFDRAHOYHANCNFSM4J6DO3VQ.

rsdunlapiv commented 4 years ago

@climbfuji greatly appreciated!

climbfuji commented 4 years ago

FYI, I am getting an error when building UFS_UTILS (related to ESMF). Hopefully the last little piece to fix.

climbfuji commented 4 years ago

Ok, I got something for you to test. Question is how you would like to load/setup the environment. The NCEPLIBS are installed in

/scratch1/BMC/gmtb/software/NCEPlibs-ufs-public/intel-18.0.5.274/impi-2018.0.4

To use them, you must do the following (trying to be as close as possible to the modulefiles/hera.intel/fv3 setup):

module load intel/18.0.5.274
module load impi/2018.0.4
module load netcdf/4.7.0

module use -a /scratch1/NCEPDEV/nems/emc.nemspara/soft/modulefiles
module load esmf/8.0.0

Question: do you want me to create a loadable module that sets an environment variable NCEPLIBS_INSTALL_DIR or something similar? Or do you just want to set this manually?

climbfuji commented 4 years ago

Note that the path will change once we are past the initial testing, probably to 

/scratch1/BMC/gmtb/software/NCEPlibs-ufs-public-v1.0.0/intel-18.0.5.274/impi-2018.0.4

and similar for a potential module name.

rsdunlapiv commented 4 years ago

On Cheyenne we have one module that sets appropriate env variables - it looks like this:

require("posix")
whatis("NCEPLIBS ufs_release_v1.0")
help([[
NCEPLIBS : Tools for support of ufs weather models

Software website - https://github.com/NOAA-EMC/NCEPLIBS

]])

prereq(atleast("intel","19.0.2"))
prereq(atleast("mpt", "2.19"))

local basepath = "/glade/work/jedwards/tools/NCEPLIBS/intel/19.0.2/"
local incpath = pathJoin(basepath, "include")
local libpath = pathJoin(basepath, "lib")

setenv("NEMSIO_INC", incpath)
setenv("NCEP_LIBS", basepath)
setenv("NEMSIO_LIB", pathJoin(libpath,"libnemsio_v2.2.3.a"))
setenv("BACIO_LIB4", pathJoin(libpath,"libbacio_v2.1.0_4.a"))
setenv("SP_LIBd",    pathJoin(libpath,"libsp_v2.0.2_d.a"))
setenv("W3EMC_LIBd", pathJoin(libpath,"libw3emc_v2.2.0_d.a"))
setenv("W3NCO_LIBd", pathJoin(libpath,"libw3nco_v2.0.6_d.a"))

However, on Stampede we set these manually in the CIME machine XML.

If you have time to create a module like the one above, I think that would be the preferred approach. Otherwise, I can set it up manually.

climbfuji commented 4 years ago

I'll create the module. But I will not use

prereq(atleast("intel","19.0.2"))

The libraries have been complied with exactly one version of the Intel compiler and MPI library, and that is what needs to be used. Unknown problems lurking everywhere.

climbfuji commented 4 years ago

Would it be possible to use NCEPLIBS_DIR instead of NCEP_LIBS? It doesn't really matter, just asking.

rsdunlapiv commented 4 years ago

@jedwards4b and @uturuncoglu would have to say whether this would cause any issues - but I would say that now is the time to get this right, and your name seems better and more consistent with other usage.

uturuncoglu commented 4 years ago

@climbfuji It is better to use NCEP_LIBS because we are using it inside of chgres and NCEP post template scripts.

uturuncoglu commented 4 years ago

If we decide to use it as NCEPLIBS_DIR then we need to make mods in machine.xml file and the template scripts.

climbfuji commented 4 years ago

@uturuncoglu I am not sure if this is good. As Rocky said, we should get this right now. For the moment, I will set both NCEPLIBS_DIR and NCEP_LIBS, but with the understanding that we will discuss this and correct what is needed within the next few days.

In addition to the module load section above, please try the following:

module use -a /scratch1/BMC/gmtb/software/modulefiles/intel-18.0.5.274/impi-2018.0.4
module load NCEPlibs-ufs-public/1.0.0.beta01

This sets:

# Set environment variables
set NCEPLIBS_DIR /scratch1/BMC/gmtb/software/NCEPlibs-ufs-public/intel-18.0.5.274/impi-2018.0.4
setenv NCEP_LIBS    ${NCEPLIBS_DIR}
setenv NCEPLIBS_DIR ${NCEPLIBS_DIR}
setenv NEMSIO_INC   ${NCEPLIBS_DIR}/include
setenv NEMSIO_LIB   ${NCEPLIBS_DIR}/lib/libnemsio_v2.2.3.a
setenv BACIO_LIB4   ${NCEPLIBS_DIR}/lib/libbacio_v2.1.0_4.a
setenv SP_LIBd      ${NCEPLIBS_DIR}/lib/libsp_v2.0.2_d.a
setenv W3EMC_LIBd   ${NCEPLIBS_DIR}/lib/libw3emc_v2.2.0_d.a
setenv W3NCO_LIBd   ${NCEPLIBS_DIR}/lib/libw3nco_v2.0.6_d.a
climbfuji commented 4 years ago

Fixed and I was able to build NCEPLIBS on Hera

@kgerheiser @mark-a-potts I was using the "official" installations of netCDF and ESMF to build NCEPLIBS. There was one problem after the successful build, namely that the rpath for some of the shared libraries (libsz and some - but not all - of the hdf5 libraries) wasn't set. This caused chgres_cube.exe and ncep-post.exe to fail (cannot find library ...).

I fixed this "manually" using patchelf and a little script to facilitate the task. See

/scratch1/BMC/gmtb/software/NCEPlibs-ufs-public/intel-18.0.5.274/impi-2018.0.4/src/installsteps.txt

and

/scratch1/BMC/gmtb/software/NCEPlibs-ufs-public/intel-18.0.5.274/impi-2018.0.4/src/fix_rpath.sh
uturuncoglu commented 4 years ago

As @rsdunlapiv said, we could make it right at this point and set it as NCEPLIBS_DIR. What do you think @jedwards4b? It could be also possible to map NCEPLIBS_DIR to NCEP_LIBS in the machine file. So, it is not a problem in CIME side.

rsdunlapiv commented 4 years ago

I suggest we go with NCEPLIBS_DIR in the long term. If it is too much to change the scripts now, let's just stick with NCEP_LIBS and we can fix this up on a future release of CIME.

climbfuji commented 4 years ago

If NCEPLIBS_DIR is the long term way to go, then I suggest making the change for this release. But not today for testing, at some point in the next days or week. For now I leave both versions in the module.

rsdunlapiv commented 4 years ago

@jedwards4b and @uturuncoglu can we provide any updates to the testing spreadsheet? https://docs.google.com/spreadsheets/d/1DnIDqZZ4toXmUmHxAs98STV19BO8TvYt1d6gkSjodrU/edit#gid=0

arunchawla-NOAA commented 4 years ago

@rsdunlapiv Does this issue need to remain open?

jedwards4b commented 4 years ago

No, it is complete.

On Thu, Feb 6, 2020 at 7:16 PM arun chawla notifications@github.com wrote:

@rsdunlapiv https://github.com/rsdunlapiv Does this issue need to remain open?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/43?email_source=notifications&email_token=ABOXUGAJWFFOK62JBIFU6Q3RBS76HA5CNFSM4J6DO3V2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELBPNOI#issuecomment-583202489, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOXUGFKLUVEFM56JJI5BU3RBS76HANCNFSM4J6DO3VQ .

-- Jim Edwards

CESM Software Engineer National Center for Atmospheric Research Boulder, CO