search

ESCOMP / CTSM

Community Terrestrial Systems Model (includes the Community Land Model of CESM)
http://www.cesm.ucar.edu/models/cesm2.0/land/
Other
305 stars 307 forks source link

Update submodule tags to pass runoff from CISM to ROF as per issue #2590 #2605

Closed slevis-lmwg closed 2 months ago

slevis-lmwg commented 3 months ago

Description of changes

Details in #2590

Specific notes

Contributors other than yourself, if any: @ekluzek @mvertens @jedwards4b @billsacks @Katetc

CTSM Issues Fixed (include github issue #): Fixes #2590

Are answers expected to change (and if so in what way)? Yes, due to change in default behavior

Any User Interface Changes (namelist or namelist defaults changes)? Default behavior has changed as explained in #2590

Does this create a need to change or add documentation? Did you do so? Probably, but I did not.

slevis-lmwg commented 3 months ago

From the TODO list in the issue Most of aux_clm and ctsm_sci is failing for now.

UPDATE: Trying aux_clm on izumi FAIL, so try bisecting between the latest tags that I used versus the branch hashes that I pointed to when the tests passed

slevis-lmwg commented 2 months ago

Bisecting between my passing and failing aux_clm test-suites:

slevis-lmwg commented 2 months ago

aux_clm on izumi now OK ./cs.status.fails | grep -v 'otherwise bit-for-b' | grep -v NLCOMP | grep -v 'EXPECTED FAILURE' | grep -v PASS

on derecho aux_clm tests_0703-123700de ./cs.status.fails | grep -v 'wise bit-for-bit' | grep -v NLCOMP | grep -v 'EXPECTED FAILURE' | grep -v PASS

    FAIL ERS_Ly3_P64x2.f10_f10_mg37.IHistClm50BgcCropG.derecho_intel.clm-cropMonthOutput BASELINE cismtorof.n01_ctsm5.2.007: DIFF
    FAIL SMS_Lm37.f10_f10_mg37.I1850Clm50SpG.derecho_intel.clm-glcMEC_long BASELINE cismtorof.n01_ctsm5.2.007: DIFF

In the first of these two, the largest NORMALIZED DIFF is ERS_Ly3_P64x2.f10_f10_mg37.IHistClm50BgcCropG.derecho_intel.clm-cropMonthOutput.C.0703-123700de_int.clm2.h0.1852-12.nc.cprnc.out: RMS ERRH2OSNO 2.5269E-17 NORMALIZED 7.0110E-02 And in the latter, the largest NORMALIZED DIFF is SMS_Lm37.f10_f10_mg37.I1850Clm50SpG.derecho_intel.clm-glcMEC_long.C.0703-123700de_int.cism.gris.h.0004-01-01-00000.nc.cprnc.out: RMS calving_rate 2.2858E-14 NORMALIZED 7.6445E-12 I interpret both as roundoff diffs that have grown over time.

ctsm_sci tests_0703-123641de ./cs.status.fails |grep -v NLCOMP|grep -v 'EXPECTED FAILURE'|grep -v 'wise bit-for-bit'

    FAIL ERP_D_Ld5.f09_g17.I2000Clm50Vic.derecho_intel.clm-vrtlay BASELINE ctsm_sci-ctsm5.2.007: DIFF
    FAIL ERP_Ld5.f09_g17.I2000Clm50Vic.derecho_intel.clm-vrtlay BASELINE ctsm_sci-ctsm5.2.007: DIFF
    FAIL SMS_D_Ld3_PS.f09_g17.I1850Clm60SpNoAnthro.derecho_intel.clm-decStart1851_noinitial BASELINE ctsm_sci-ctsm5.2.007: DIFF
    FAIL SMS_Ld1.f19_g17.I2000Clm50Vic.derecho_intel.clm-default BASELINE ctsm_sci-ctsm5.2.007: DIFF
    FAIL SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial BASELINE ctsm_sci-ctsm5.2.007: DIFF

In the last of these, I see larger diffs including in variables that I would not have expected initially. This is the only Bgc case here. Should I expect changes in CH4 variables as a result of the changes in river flow @swensosc?

SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS CH4PROD                          1.4435E-12            NORMALIZED  7.6136E-03
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS CH4_SURF_AERE_SAT                6.8893E-14            NORMALIZED  9.1275E-02
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS CH4_SURF_DIFF_SAT                5.9901E-12            NORMALIZED  4.6836E-02
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS CH4_SURF_EBUL_SAT                1.5077E-11            NORMALIZED  5.6659E-02
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS FCH4                             3.5318E-16            NORMALIZED  2.9839E-04
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS FCH4TOCO2                        2.1029E-13            NORMALIZED  3.5289E-04
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS FCH4_DFSAT                       3.5660E-16            NORMALIZED  5.2060E+00
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS FINUNDATED                       6.8805E-05            NORMALIZED  8.7284E-03
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS NEM                              1.3188E-12            NORMALIZED  2.6262E-03
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS TOTCOLCH4                        8.2472E-08            NORMALIZED  2.3369E-04
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS TWS                              1.5401E+00            NORMALIZED  1.0340E-04
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS VOLR                             2.0453E+07            NORMALIZED  2.1061E-01
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS VOLRMCH                          2.0453E+07            NORMALIZED  2.1061E-01
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.clm2.h0.1852-01-02-00000.nc.cprnc.out: RMS CONC_O2_SAT                      4.8355E-06            NORMALIZED  3.8462E-03
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.cpl.hi.1852-01-02-00000.nc.cprnc.out: RMS lndExp_Flrr_volr                 1.0665E-03            NORMALIZED  4.0627E-01
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.cpl.hi.1852-01-02-00000.nc.cprnc.out: RMS lndExp_Flrr_volrmch              1.0665E-03            NORMALIZED  4.0627E-01
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.cpl.hi.1852-01-02-00000.nc.cprnc.out: RMS rofImp_Flrr_volr                 1.5704E-03            NORMALIZED  5.9480E-01
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.cpl.hi.1852-01-02-00000.nc.cprnc.out: RMS rofImp_Flrr_volrmch              1.5704E-03            NORMALIZED  5.9480E-01
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.cpl.hi.1852-01-02-00000.nc.cprnc.out: RMS rofImp_Forr_rofl                 7.2396E-06            NORMALIZED  1.0973E+01
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.rtm.h0.1851-12.nc.cprnc.out: RMS QCHANR                           3.5056E+01            NORMALIZED  3.3240E-01
SMS_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthro.derecho_intel.clm-decStart1851_noinitial.C.0703-123641de_int.rtm.h0.1851-12.nc.cprnc.out: RMS QCHOCNR                          7.3298E+00            NORMALIZED  2.3645E+00
slevis-lmwg commented 2 months ago

@swensosc verified by email that these diffs seem reasonable. Sean wrote: I think you would see changes in methane due to the fact that the riley et al. method for calculating inundation uses a regression against TWS to determine the inundated fraction, which is an input to the methane model. So if VOLR is changed, TWS would be changed, and thus methane would see a perturbed inundated area.

slevis-lmwg commented 2 months ago

testing on derecho PASS ./build-namelist_test.pl (still passes after the fix mentioned below) PASS make black and make lint (still passes after the fix mentioned below) PASS ./run_ctsm_py_tests -u and ./run_ctsm_py_tests -s (still passes after the fix mentioned below)

./run_sys_tests -s aux_clm -c ctsm5.2.014 -g ctsm5.2.015 OK on derecho FAIL 1 test on izumi that I have fixed.

I will rerun aux_clm on derecho and izumi to be safe.

slevis-lmwg commented 2 months ago

@jedwards4b I updated to ctsm5.2.014 and submitted tests as posted above. If all goes smoothly, I should merge this PR next.

First I wanted your input on something that seems concerning to me about ccs_config: ./bin/git-fleximod update (with or without --force) returns...

s  ccs_config ccs_config_cesm0.0.113 69a958581ecd2d32ee9cb1c38bcd3847b8b920bf is out of sync with .gitmodules ccs_config_cesm1.0.0

ccs_config updated to ccs_config_cesm1.0.0

and then status returns...

s  ccs_config ccs_config_cesm0.0.113 69a958581ecd2d32ee9cb1c38bcd3847b8b920bf is out of sync with .gitmodules ccs_config_cesm1.0.0

and git describe returns... ccs_config_cesm0.0.113 while .gitmodules says... fxtag = ccs_config_cesm1.0.0 What do you think?

slevis-lmwg commented 2 months ago

@jedwards4b I tried git diff ccs_config_cesm1.0.0 ccs_config_cesm0.0.113 and got no difference, so I will assume that ccs_config is acceptable for the purposes of this PR's testing.

slevis-lmwg commented 2 months ago

Thank you @jedwards4b

aux_clm results ./cs.status.fails | grep -v 'ise bit-for-b' | grep -v NLCOMP|grep -v PASS | grep -v '14: DIF' OK izumi OK derecho Last TODO before merge: I want to double check the nature of the DIFFs from baseline.

slevis-lmwg commented 2 months ago

These differences seem out of the ordinary:

SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdDryDepSatPhen.GC.0719-175606de_gnu/SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdDryDepSatPhen.GC.0719-175606de_gnu.clm2.h0.2000-01-01-00000.nc.cprnc.out: RMS FATES_TRANSITION_MATRIX_LULU       Infinity            NORMALIZED    Infinity
SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdMeganSatPhen.GC.0719-175606de_gnu/SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdMeganSatPhen.GC.0719-175606de_gnu.clm2.h0.2000-01-01-00000.nc.cprnc.out: RMS FATES_TRANSITION_MATRIX_LULU       Infinity            NORMALIZED    Infinity

@glemieux are these new tests that came with your recent PR? Does this concern you?

glemieux commented 2 months ago

These differences seem out of the ordinary:

SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdDryDepSatPhen.GC.0719-175606de_gnu/SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdDryDepSatPhen.GC.0719-175606de_gnu.clm2.h0.2000-01-01-00000.nc.cprnc.out: RMS FATES_TRANSITION_MATRIX_LULU       Infinity            NORMALIZED    Infinity
SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdMeganSatPhen.GC.0719-175606de_gnu/SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdMeganSatPhen.GC.0719-175606de_gnu.clm2.h0.2000-01-01-00000.nc.cprnc.out: RMS FATES_TRANSITION_MATRIX_LULU       Infinity            NORMALIZED    Infinity

@glemieux are these new tests that came with your recent PR? Does this concern you?

@slevis-lmwg can you point me to the test folder? I'd like to check out the cprnc output.

slevis-lmwg commented 2 months ago

@glemieux please look here /glade/work/slevis/git_externals/cism2mosart/tests_0719-175606de

jedwards4b commented 2 months ago

There is a permissions problem for your baseline directory.
ls: cannot open directory '/glade/campaign/cgd/tss/ctsm_baselines/ctsm5.2.014/': Permission denied

slevis-lmwg commented 2 months ago

@jedwards4b I updated permissions for that one and will go through and update for my other ones, too. LMK if you still have trouble accessing.

jedwards4b commented 2 months ago

Thank you for fixing that - the values for the field FATES_TRANSITION_MATRIX_LULU in both the baseline and the case look like garbage to me. Can someone confirm these values are expected?

slevis-lmwg commented 2 months ago

@glemieux is taking a look.

glemieux commented 2 months ago

Yep, those cases are definitely garbage, but they've been that way for these two test since that variable was introduced in ctsm5.2.013. I missed this during testing. Those values should be zero'd out. I'm not sure why the FatesColdSatPhen correctly zeros FATES_TRANSITION_MATRIX_LULU, but FatesColdDryDepSatPhen and FatesColdMeganSatPhen does not.

I'd say this is ok as is to go forward, but an issue should be created to fix this as it will likely flag as a DIFF again in the future given the randomness of the garbage.

@slevis-lmwg and @jedwards4b how does that sound to y'all?

slevis-lmwg commented 2 months ago

I'm fine with that. I can open the issue and you can add to it as you see fit @glemieux Thanks for looking into this.