search

ufs-community / UFS_UTILS

Utilities for the NCEP models.
Other
21 stars 104 forks source link

Chgres_cube regression test failure caused by difference in module files between develop and SRW app #234

Closed LarissaReames-NOAA closed 1 year ago

LarissaReames-NOAA commented 3 years ago

When chgres_cube is compiled and tested with the module files included in the develop branch, the regression tests pass as expected. However, when it is compiled with the module files in the SRW app, the tests fail on tile 6 of the surface fields with very small magnitude errors similar to 

sheleg   /         3  4.26326e-14 7.10543e-14 -1.42109e-14  2.84217e-14 4.26326e-14  1.42109e-14 2.46139e-14
snwdph   /         1 -2.27374e-13 2.27374e-13 -2.27374e-13 -2.27374e-13           0 -2.27374e-13           0

To see all of the output on Jet, view files

/home/Larissa.Reames/UFS_UTILS_release/reg_tests/chgres_cube/regression.log (release branch modules)
/home/Larissa.Reames/UFS_UTILS_release/reg_tests/chgres_cube/regression.log.devmod (develop branch modules)

This problem started when the baseline files were re-created on November 19th and occurs on both Jet and Hera. My guess is that the difference in ESMF versions is perhaps causing these issues. I suggest we maintain different regression tests for the release branch and the develop branch as long as they are using different library versions.

climbfuji commented 3 years ago

Of course we keep separate baselines for the ufs-weather-model for release/public-v2 and develop, and I would expect the same for UFS_UTILS and any other library/tool.

GeorgeGayno-NOAA commented 3 years ago

I created a set of baseline data for Hera (using 0b68195). For now, it is stored here: /scratch1/NCEPDEV/da/George.Gayno/noscrub/reg_tests/chgres_cube.public.v2

LarissaReames-NOAA commented 3 years ago

@GeorgeGayno-NOAA Great, the regression tests in develop are now passing with the executable built from the app's modules as well, so it appears this has solved the problem. What are your thoughts on where we store these permanently? The easiest way, in my mind, would be to have the directory as you do now (reg_tests/chgres_cube.public.v2) and with the fix and input_data files linked in from reg_tests/chgres_cube. Then in each driver script in the release/public-v2 branch I'd make that one change to HOMEreg.

GeorgeGayno-NOAA commented 3 years ago

@GeorgeGayno-NOAA Great, the regression tests in develop are now passing with the executable built from the app's modules as well, so it appears this has solved the problem. What are your thoughts on where we store these permanently? The easiest way, in my mind, would be to have the directory as you do now (reg_tests/chgres_cube.public.v2) and with the fix and input_data files linked in from reg_tests/chgres_cube. Then in each driver script in the release/public-v2 branch I'd make that one change to HOMEreg.

I don't mind storing the baseline files in my own space. But maybe this could be discussed at the next SRW App Code Focus meeting.

JeffBeck-NOAA commented 3 years ago

@GeorgeGayno-NOAA @LarissaReames-NOAA, I'm fine with this proposal. I haven't proposed any further SRW App Code Focus team meetings at the moment, since the code freeze is essentially complete (except for a few minor details in ufs-srweather-app and regional_workflow that I am working on with Gerard and Mike). I can certainly arrange for a meeting, but I'm wondering if a targeted email to interested parties would work instead?

JeffBeck-NOAA commented 3 years ago

@GeorgeGayno-NOAA, just wanted to check with you to see if you were able to create two directories with the baselines specific to each branch (develop and release/public-v2)? For now, I think that's a satisfactory solution.

GeorgeGayno-NOAA commented 3 years ago

@GeorgeGayno-NOAA, just wanted to check with you to see if you were able to create two directories with the baselines specific to each branch (develop and release/public-v2)? For now, I think that's a satisfactory solution.

I have the two directories on Hera. What other machines will be supported? Also, the regression test scripts in the public release branch will need to point to the new location via $HOMEreg.

JeffBeck-NOAA commented 3 years ago

@GeorgeGayno-NOAA, we should also include Jet, Orion, Cheyenne, WCOSS, and Odin, at least. Regarding the change to $HOMEreg, I assume we can make that change in the release branch scripts and push a PR, correct, @LarissaReames-NOAA?

GeorgeGayno-NOAA commented 3 years ago

@GeorgeGayno-NOAA, we should also include Jet, Orion, Cheyenne, WCOSS, and Odin, at least. Regarding the change to $HOMEreg, I assume we can make that change in the release branch scripts and push a PR, correct, @LarissaReames-NOAA?

I updated the baseline data on Jet, Orion, Hera and WCOSS. I don't have access to the other machines. To use the data, adjust $HOMEreg to point to ./chgres_cube.public.v2.

GeorgeGayno-NOAA commented 2 years ago

@JeffBeck-NOAA and @LarissaReames-NOAA Can we close this?