Add cryo terms to coupler budget

[darincomeau]

Adds five new o2x coupling fields and terms to coupler budget for polar configurations:

wism - water from ice shelf basal melting (either prognostic or data) wrrof - water removed from Antarctica liquid runoff wriof - water removed from Antarctica solid runoff

hism - heat from ice shelf basal melting hriof - heat from removed ice runoff

[darincomeau]

Results from the last month of one year tests.

ne30pg2_ECwISC30to60E2r1.WCYCL1850:

(seq_diag_print_mct) NET WATER BUDGET (kg/m2s*1e6): period = all_time: date =     20101     0
                           atm            lnd            rof            ocn         ice nh         ice sh            glc        *SUM*
         wfreeze     0.00000000     0.00000000     0.00000000    -0.14447987     0.07808967     0.06639020     0.00000000     0.00000000
           wmelt     0.00000000     0.00000000     0.00000000     0.56578336    -0.56105289    -0.00474712     0.00000000    -0.00001666
           wrain   -32.31931275     6.58372094     0.00000000    25.67277765     0.04991955     0.01230397     0.00000000    -0.00059064
           wsnow    -2.13925165     0.83594827     0.00000000     0.95016920     0.13399754     0.21911914     0.00000000    -0.00001750
           wevap    34.46006843    -4.38902620     0.00000000   -30.04343738    -0.01192597    -0.01577898     0.00000000    -0.00010010
         wrunoff     0.00000000    -0.60896813     0.06726314     0.54133451     0.00000000     0.00000000     0.00000000    -0.00037048
         wfrzrof     0.00000000    -0.20327073     0.00000000     0.20325413     0.00000000     0.00000000     0.00000000    -0.00001660
           wberg     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000
            wism     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000
           wrrof     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000
           wriof     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000
          wirrig     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000
           *SUM*     0.00150403     2.21840414     0.06726314    -2.25459840    -0.31097210     0.27728721     0.00000000    -0.00111198

ne30pg2_ECwISC30to60E2r1.CRYO1850-DISMF:

(seq_diag_print_mct) NET WATER BUDGET (kg/m2s*1e6): period = all_time: date =     20101     0
                           atm            lnd            rof            ocn         ice nh         ice sh            glc        *SUM*
         wfreeze     0.00000000     0.00000000     0.00000000    -0.10856025     0.07964831     0.02891194     0.00000000    -0.00000000
           wmelt     0.00000000     0.00000000     0.00000000     0.56418650    -0.54820973    -0.01596599     0.00000000     0.00001078
           wrain   -32.10275677     6.38852996     0.00000000    25.65772076     0.04453218     0.01147925     0.00000000    -0.00049462
           wsnow    -2.13062436     0.83462355     0.00000000     0.95319482     0.13564092     0.20717841     0.00000000     0.00001335
           wevap    34.22371427    -4.36568440     0.00000000   -29.83287799    -0.01057680    -0.01467655     0.00000000    -0.00010146
         wrunoff     0.00000000    -0.59464619     0.06129152     0.53293445     0.00000000     0.00000000     0.00000000    -0.00042022
         wfrzrof     0.00000000    -0.21128857     0.00000000     0.21127484     0.00000000     0.00000000     0.00000000    -0.00001372
           wberg     0.00000000     0.00000000     0.00000000     0.07937360     0.00000000     0.00000000     0.00000000     0.07937360
            wism     0.00000000     0.00000000     0.00000000     0.03377146     0.00000000     0.00000000     0.00000000     0.03377146
           wrrof     0.00000000     0.00000000     0.00000000    -0.01411851     0.00000000     0.00000000     0.00000000    -0.01411851
           wriof     0.00000000     0.00000000     0.00000000    -0.17140508     0.00000000     0.00000000     0.00000000    -0.17140508
          wirrig     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000
           *SUM*    -0.00966686     2.05153436     0.06129152    -2.09450538    -0.29896512     0.21692706     0.00000000    -0.07338442 

ne30pg2_ECwISC30to60E2r1.CRYO1850:

(seq_diag_print_mct) NET WATER BUDGET (kg/m2s*1e6): period = all_time: date =     20101     0
                           atm            lnd            rof            ocn         ice nh         ice sh            glc        *SUM*
         wfreeze     0.00000000     0.00000000     0.00000000    -0.11310092     0.08161947     0.03148145     0.00000000    -0.00000000
           wmelt     0.00000000     0.00000000     0.00000000     0.61626948    -0.58662693    -0.02965021     0.00000000    -0.00000766
           wrain   -32.14911941     6.56787763     0.00000000    25.51340667     0.05426512     0.01304402     0.00000000    -0.00052597
           wsnow    -2.16887503     0.84515116     0.00000000     0.96718653     0.12969461     0.22683606     0.00000000    -0.00000666
           wevap    34.32869776    -4.37683964     0.00000000   -29.92584677    -0.01067165    -0.01544155     0.00000000    -0.00010186
         wrunoff     0.00000000    -0.60071222     0.06399359     0.53630126     0.00000000     0.00000000     0.00000000    -0.00041738
         wfrzrof     0.00000000    -0.20471793     0.00000000     0.20465379     0.00000000     0.00000000     0.00000000    -0.00006414
           wberg     0.00000000     0.00000000     0.00000000     0.07937360     0.00000000     0.00000000     0.00000000     0.07937360
            wism     0.00000000     0.00000000     0.00000000     0.07718944     0.00000000     0.00000000     0.00000000     0.07718944
           wrrof     0.00000000     0.00000000     0.00000000    -0.01619780     0.00000000     0.00000000     0.00000000    -0.01619780
           wriof     0.00000000     0.00000000     0.00000000    -0.16214656     0.00000000     0.00000000     0.00000000    -0.16214656
          wirrig     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000     0.00000000
           *SUM*     0.01070332     2.23075900     0.06399359    -2.22291127    -0.33171938     0.22626976     0.00000000    -0.02290498

[darincomeau]

I've posted a single month of the water coupler budget, but for the polar runs, just noting that looking at a single month can be a bit misleading, since the ice-shelf/berg melt has the opposite seasonal cycle as the removed runoff terms.

Ideally we would like to see, over the course of the year, that (wism + wberg) \approx (wrrof + wriof).

EDIT: the tables above are the all-time sums over the 1 year simulations.

[xylar]

@darincomeau, I don't really speak coupler yet (though I'm keen to get on Duolingo or wherever and learn some!). Is the coupler budget output just the accumulated change in whatever field over the given period (e.g. a month?), as opposed to accumulated over the simulation?

[xylar]

By the way, thanks so much for getting this going! I'm realizing how far over my head this would have been.

[darincomeau]

Is the coupler budget output just the accumulated change in whatever field over the given period (e.g. a month?), as opposed to accumulated over the simulation?

Good question; I thought they were monthly sums, but given it says 'period = all_time:', perhaps they are accumulations over the entire simulation, which would be preferable.

[jonbob]

@xylar and @darincomeau -- by default, the coupler budgets are output monthly, annually, and all-time. But this is controlled by flags which also allow daily and instantaneous options

[xylar]

@jonbob, I guess my question was whether the field going in should be removedRiverRunoffFlux, etc. as @darincomeau has it or accumulatedRemovedRiverRunoffFlux, etc. from the analysis member that adds those up in MPAS-Ocean. It sounds like @darincomeau is probably doing things right and the coupler itself takes care of accumulating everything over the desired period?

[darincomeau]

It sounds like @darincomeau is probably doing things right and the coupler itself takes care of accumulating everything over the desired period?

I was also wondering about accumulation, in particular when the mpas-ocean timestep is shorter than the coupling interval.

[xylar]

Exactly!

[xylar]

@darincomeau, I think we probably need to at the very lease accumulate the field over the coupling interval like happens with other fields passed to the coupler.

[darincomeau]

Ok, I'll try to find an analog, but if you have one in mind, I'll take it as a starting point.

[xylar]

https://github.com/E3SM-Project/E3SM/blob/master/components/mpas-ocean/src/shared/mpas_ocn_time_average_coupled.F

[jonbob]

Sorry if I misunderstood. But you are correct -- the coupler will take care of accumulations in general but the ocean needs to do its own accumulation over the coupling interval

[xylar]

@jonbob, thanks, that clears things up.

[darincomeau]

https://github.com/E3SM-Project/E3SM/blob/master/components/mpas-ocean/src/shared/mpas_ocn_time_average_coupled.F

Thanks @xylar - looks like this is an important step I missed!

[xylar]

Maybe use avgTotalFreshWaterTemperatureFlux as your field to imitate?

https://github.com/E3SM-Project/E3SM/blob/3e727e0fbbae951added95895482e5a0da36069a/components/mpas-ocean/src/Registry.xml#L3689-L3691

[darincomeau]

With that last commit, the results for ne30pg2_ECwISC30to60E2r1.CRYO1850 are exactly the same. I then realized for that resolution the ocean timestep is the same as the coupling interval. So in a way I guess that's good confirmation, but I'll test with a SORRM mesh for a case where the ocean timestep is less than the coupling interval. Data ISMF are not being accounted for however, I'm not sure if the variable is not yet assigned when trying to be used in mpas_ocn_time_averaged.F, or something else.

[xylar]

@darincomeau, one quick thing before I forget -- your branch name shouldn't have capital letters according to https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/2523172/Branch+Tag+and+Version+name+conventions. So please rename it to all lowercase before we make an E3SM PR.

[xylar]

I tried to test this on Chrysalis but, once again, we're out of space.

[darincomeau]

@darincomeau, one quick thing before I forget -- your branch name shouldn't have capital letters according to https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/2523172/Branch+Tag+and+Version+name+conventions. So please rename it to all lowercase before we make an E3SM PR.

The "OD" in the branch name here is because it's off the head of Ocean Discussions. The original branch is here: https://github.com/darincomeau/E3SM/tree/cpl/add-cryo-budgets

That original branch is off the head of recent master, so trying to open a PR into this repo was trying to bring in a bunch of unrelated commits, so I just made this second temporary branch and have been cherry-picking the commits over.

[darincomeau]

I tried to test this on Chrysalis but, once again, we're out of space.

Well thanks for trying! I'll get this branch updated so that those obvious errors aren't there (but still may not be working as intended).

[xylar]

That original branch is off the head of recent master, so trying to open a PR into this repo was trying to bring in a bunch of unrelated commits, so I just made this second temporary branch and have been cherry-picking the commits over.

I see. Just ask us to update master here on E3SM-Ocean-Discussion when that happens. It's better to use the same branch.

Once master here is updated, you can switch the target branch to alternate and then back to master and the unrelated commits should disappear.

But don't worry about it for this PR, since you can't change branches for an open PR anyway.

[darincomeau]

Yep agreed, I meant to bring up updating here in last week's meeting but forgot.

[xylar]

@darincomeau, to debug, I would suggest making a new ocean stream that gets written out every time step and use it to write out all of the variables involved in DISMF (both new and old) to figure out which are zeros and which aren't. Then, just run for 1 day or something like that so you don't get a crazy amount of output. I'll try this myself but won't have time until tomorrow.

[xylar]

I tried to run a CRYO1850-DISMF.ne30pg2_IcoswISC30E3r5 configuration on Chrysalis (both with this branch and with master) and I'm getting seg faults:

 60:  0 0x0000000000012b20 .annobin_sigaction.c()  sigaction.c:0
 60:  1 0x000000000633b94c esmf_clockmod_mp_esmf_clockget_()  /gpfs/fs1/home/ac.xylar/e3sm_work/E3SM/master/share/esmf_wrf_timemgr/ESMF_ClockMod.F90:585
 60:  2 0x0000000005e51a83 mpas_timekeeping_mp_mpas_get_clock_time_()  /lcrc/group/e3sm/ac.xylar/scratch/chrys/20240205.CRYO1850-DISMF.ne30pg2_IcoswISC30E3r5.chrysalis/bld/cmake-bld/framework/mpas_timekeeping.f90:453
 60:  3 0x0000000005f52162 mpas_forcing_mp_get_initial_forcing_times_()  /lcrc/group/e3sm/ac.xylar/scratch/chrys/20240205.CRYO1850-DISMF.ne30pg2_IcoswISC30E3r5.chrysalis/bld/cmake-bld/framework/mpas_forcing.f90:805
 60:  4 0x0000000005f4d49b mpas_forcing_mp_create_new_forcing_stream_()  /lcrc/group/e3sm/ac.xylar/scratch/chrys/20240205.CRYO1850-DISMF.ne30pg2_IcoswISC30E3r5.chrysalis/bld/cmake-bld/framework/mpas_forcing.f90:512
 60:  5 0x0000000005f52a8e mpas_forcing_mp_mpas_forcing_init_field_()  /lcrc/group/e3sm/ac.xylar/scratch/chrys/20240205.CRYO1850-DISMF.ne30pg2_IcoswISC30E3r5.chrysalis/bld/cmake-bld/framework/mpas_forcing.f90:352
 60:  6 0x000000000480b86b seaice_forcing_mp_seaice_forcing_init_()  /lcrc/group/e3sm/ac.xylar/scratch/chrys/20240205.CRYO1850-DISMF.ne30pg2_IcoswISC30E3r5.chrysalis/bld/cmake-bld/core_seaice/shared/mpas_seaice_forcing.f90:2973
 60:  7 0x0000000004831f73 seaice_initialize_mp_seaice_init_()  /lcrc/group/e3sm/ac.xylar/scratch/chrys/20240205.CRYO1850-DISMF.ne30pg2_IcoswISC30E3r5.chrysalis/bld/cmake-bld/core_seaice/shared/mpas_seaice_initialize.f90:153
 60:  8 0x0000000004fdb809 seaice_core_mp_seaice_core_init_()  /lcrc/group/e3sm/ac.xylar/scratch/chrys/20240205.CRYO1850-DISMF.ne30pg2_IcoswISC30E3r5.chrysalis/bld/cmake-bld/core_seaice/model_forward/mpas_seaice_core.f90:251
 60:  9 0x00000000046422e3 ice_comp_mct_mp_ice_init_mct_()  /lcrc/group/e3sm/ac.xylar/scratch/chrys/20240205.CRYO1850-DISMF.ne30pg2_IcoswISC30E3r5.chrysalis/mpas-seaice/driver/ice_comp_mct.f90:632
 60: 10 0x0000000000455f14 component_mod_mp_component_init_cc_()  /gpfs/fs1/home/ac.xylar/e3sm_work/E3SM/master/driver-mct/main/component_mod.F90:257
 60: 11 0x00000000004422a6 cime_comp_mod_mp_cime_init_()  /gpfs/fs1/home/ac.xylar/e3sm_work/E3SM/master/driver-mct/main/cime_comp_mod.F90:1475
 60: 12 0x0000000000452c0a MAIN__()  /gpfs/fs1/home/ac.xylar/e3sm_work/E3SM/master/driver-mct/main/cime_driver.F90:122
 60: 13 0x00000000004280a2 main()  ???:0
 60: 14 0x0000000000023493 __libc_start_main()  ???:0
 60: 15 0x0000000000427fae _start()  ???:0

Obviously not related to this branch but it's disconcerting. @jonbob, any ideas what would cause this?

This is just a normal 5-day run where I didn't change any options at all in the case directory before setting up, building and submitting.

/home/ac.xylar/e3sm_cases/20240205.CRYO1850-DISMF.ne30pg2_IcoswISC30E3r5.chrysalis
/lcrc/group/e3sm/ac.xylar/scratch/chrys/20240205.CRYO1850-DISMF.ne30pg2_IcoswISC30E3r5.chrysalis

[darincomeau]

@xylar confirming I can reproduce that problem with SMS_P2048.ne30pg2_IcoswISC30E3r5.CRYO1850-DISMF.chrysalis_intel on master. Seg faults in ice during "Initialize forcing...".

[darincomeau]

@darincomeau, to debug, I would suggest making a new ocean stream that gets written out every time step and use it to write out all of the variables involved in DISMF (both new and old) to figure out which are zeros and which aren't. Then, just run for 1 day or something like that so you don't get a crazy amount of output.

Thanks for the suggestion @xylar . I didn't have a chance to work on this today, but I pushed changes that fix the logical error, and another that splits out the data ISMF accumulation (still doesn't work).

[darincomeau]

I'm also seeing that in the ocean conservation analysis member, with data ISMF we're getting accumulatedLandIceFlux = 0 (even after the above commit's change).

But landIceFreshwaterFlux is non-zero in the timeSeriesStatsMonthly file with data ISMF.

So, still stumped here.

[xylar]

@darincomeau, I got distracted by the fact that the CRYO1850 compset isn't working with the Icos mesh but haven't really made much progress on either that or the DISMF issue here. I did find out that a G-case with JRA 1.5 and DIB works fine with the Icos mesh so it's something weird with the B-case specifically.

[darincomeau]

No worries @xylar , I think the Icos CRYO1850 issue is more pressing since that will be what we'll want to test (well with DISMF). I'm hoping there's something simple I'm missing here, just reporting findings that I can't make sense of just yet.

[darincomeau]

BTW I've been using the following to run one month so we have both coupler budget and ocean conservation analysis member/budget: ./create_test SMS_Lm1_P2048.ne30pg2_ECwISC30to60E2r1.CRYO1850-DISMF.chrysalis_intel --walltime="00:30:00"

[xylar]

@darincomeau, that sounds good but you'll suffer from having to completely rebuild the model with each code change if you use create_test rather than create_newcase so not the easiest for debugging.

[xylar]

I'm doing a very manual process of debugging the CRYO1850 with IcoswISC30E3r5 issue. I'm making slow headway.

[darincomeau]

thanks @xylar - I'm going to try again with these 0 values this afternoon.

you'll suffer from having to completely rebuild the model with each code change if you use create_test rather than create_newcase so not the easiest for debugging.

yeah it has been slow, each change takes like 30-45 minutes to go through. But if I'm making code changes, shouldn't I want a fresh build each time?

[xylar]

But if I'm making code changes, shouldn't I want a fresh build each time?

Not typically. ./case.build has been doing a great job of only rebuilding the changed code. It's much faster.

[rljacob]

You can cd to the casedir set up by create_test and do edit-./case.build - ./case.submit.

[darincomeau]

Ok, good to know I can safely reuse the same case. thanks!

[darincomeau]

I'm actually now seeing

data:

 accumulatedLandIceFlux = 17215276.0302936 ;
}

in a CRYO1850-DISMF case, so I'm going to cross that off the list.

[xylar]

@darincomeau, great! What ended up being the problem?

[xylar]

Thanks @rljacob. That seems obvious once you say it but I hadn't thought to do that.

[xylar]

@darincomeau, I've traced the Icos-CRYO1850 problem to config_do_restart inexplicably being true even though it's false both in the namelist file and in the list of namelist options printed earlier in the log file.

[xylar]

Unless you have time to investigate more today, I'll pick up on this tomorrow. Hard to end the day without a win on this but 10 pm is really time to quit...

[darincomeau]

@darincomeau, great! What ended up being the problem?

No idea, my latest test from last night that had the same change in the ocean conservation analysis member had this:

data:

 accumulatedLandIceFlux = 0 ;
}

So, I'm going to take that as human error and must have not run what I thought I was running.

[xylar]

Okay, I figured it out!

We set config_do_restart = .true. here: https://github.com/E3SM-Project/E3SM/blob/50485d0c929a873b5935ea3c2dc1da4d236319dd/components/mpas-seaice/src/model_forward/mpas_seaice_core.F#L125 if we're doing an E3SM restart run (which we are in this config because we're using a spun-up initial condition).

But the spinup didn't include DIB so the associated forcing timers weren't in the restart file. It seems MPAS-Seaice has a bug where it can't turn on forcing that wasn't on in the beginning of the simulation. which is annoying and a problem!

@darincomeau, can you try to recruit folks to help fix this ASAP?

[xylar]

The specific problem is that all the contents of forcingGroupNames are empty strings in this loop: https://github.com/E3SM-Project/E3SM/blob/50485d0c929a873b5935ea3c2dc1da4d236319dd/components/mpas-framework/src/framework/mpas_forcing.F#L2680

[darincomeau]

Nice find @xylar! So I suppose the immediate fix is to generate a new 1 month ocn/ice ICs that include DIB.

[xylar]

So I suppose the immediate fix is to generate a new 1 month ocn/ice ICs that include DIB.

Yeah, actually, that might be the easiest thing to do. Can we have a different IC for DIB runs than for non-DIB? I don't think we can change the IC for everyone for the Icos mesh without getting ourselves in trouble.

[xylar]

Also, why has this never come up before? I guess we've never done a spin-up with cavities but without DIB, right?

[darincomeau]

Yeah it seems like this should have come up before.

Anyways, I'll make the initial condition and look into if we can add it as a separate one for compsets with active DIB.

[xylar]

Anyways, I'll make the initial condition and look into if we can add it as a separate one for compsets with active DIB.

I would think you would change this: https://github.com/E3SM-Project/E3SM/blob/master/components/mpas-seaice/cime_config/buildnml#L262-L264 to something like:

        if ice_ic_mode == 'spunup':
            if iceberg_mode == 'data':
                grid_date = '20240208'
                grid_prefix = 'mpassi.IcoswISC30E3r5.rstFromG-DIB-chrysalis'
            else:
                grid_date = '20231121'
                grid_prefix = 'mpassi.IcoswISC30E3r5.rstFromG-chrysalis'

The clock is ticking! You don't want to make my placeholder date stamp wrong, do you!