cism runoff will be now routed to ocn via mosart

[mvertens]

This PR enables CISM runoff to be routed to the ocean via mosart.

• All runoff from CISM will be routed directly to the outlet points
• New fields will be advertised in the mosart cap call fldlist_add(fldsFrRof_num, fldsFrRof, 'Forr_rofl_glc') call fldlist_add(fldsFrRof_num, fldsFrRof, 'Forr_rofi_glc')

Issues resolved:

• Resolves: #102
• Resolves: #92
• Resolves: #101

TODOs:

• [x] This PR must come in after #76 since it is built on top of those changes.
• [x] Merge in mosart1.1.01 into this branch (shouldn't change much, since this was built on top of #76)
• [x] This PR must also be accompanied by CISM, CMEPS, ccs_config PRs.
  • https://github.com/ESCOMP/CMEPS/pull/463
  • https://github.com/ESCOMP/CISM-wrapper/pull/100
  • https://github.com/ESMCI/ccs_config_cesm/pull/174
• [x] Ran mosart test suite on derecho and compared with mosart1_0_49 baseline Results are bfb (other than new fields on history files) and new baselines created mosart1.1.2
• [x] Some validation has been carried out. See https://github.com/ESCOMP/CISM-wrapper/issues/64. Determine if more validation is needed. @billsacks @katec - are we okay with the current state of the validation?

[mvertens]

I recommend removing all direct references to mpi and using ESMF_VM calls instead - this future proofs the code a little in case the mpi library is replaced with some other comm library on some future hardware - if all our calls are through the esmf layer then only esmf needs to be updated.

I have concerns about this (also listed below).

• ESMF_VMAllReduce only accepts one-dimensional arrays. tmp_in and tmp_glob are 2d-arrays. So copies will need to be made and 6 calls will need to be done and then copies need to be made. Frankly, I am worried that moving to only ESMF MPI communication will make the code harder to understand since it really supports only a very small subset of MPI communications. And I think it is not a good idea to mix ESMF calls in the code with MPI calls. Happy to talk about this some more.
• Using ESMF_VMBroadcast - originally I thought this was a good idea. Looking at this some more I found that ESMF_VMBroadcast only accepts arrays. That means that an array of length 1 needs to be created and then a copy of finidat needs to be made to that and then broadcast and then copied back. I think this will make the code more complicated and harder to read.

I have spoken with @jedwards4b - and created the issue https://github.com/ESCOMP/MOSART/issues/100. I will not be implementing these changes in this PR.

[mvertens]

@slevis-lmwg - I still need to run the mosart test suite and make sure that everything is bfb. So no hurry to approve just yet.

[slevis-lmwg]

@slevis-lmwg - I still need to run the mosart test suite and make sure that everything is bfb. So no hurry to approve just yet.

@mvertens will you merge mosart1.1.01 to this PR or would you like me to do it and, if the latter, then before or after you run the test suite?

[mvertens]

@slevis-lmwg - I'll merge mosart1.1.0.1 to this PR before I run the test suite.

[mvertens]

@slevis-lmwg - I think this is ready for merging based on all the tests I've done. @jedwards4b - do you want to review again?

[mvertens]

@billsacks @Katetc - are you both okay with the current level of validation?

[billsacks]

Yes, I'm satisfied with the current level of validation. Thank you very much for your work on this @mvertens !!!

[slevis-lmwg]

I have checked out this branch and submitted the test-suites on izumi and derecho: ./run_sys_tests -s mosart -c mosart1.1.01_ctsm5.2.007 -g mosart1.1.02_ctsm5.2.007

[slevis-lmwg]

Oh, right, and i have confirmed that we need to complete https://github.com/ESCOMP/CTSM/issues/2590 for the tests to pass.

[jedwards4b]

I think that there is going to be a bit of a chicken and egg problem here - you will need to use some PR branches to test some other PR's.

[mvertens]

@ekluzek @billsacks @slevis-lmwg - I have addressed the issues in this PR except for addressing refactoring the use of 'subname' and reintroducing empty routines whose presence were confusing.

[billsacks]

It looks like @mvertens has addressed all of the critical points here; if so, it would be great if any non-critical pieces can be deferred to a follow-up PR. @ekluzek and @slevis-lmwg are there any other important needs that you see that need to be addressed before this can be brought in?

[ekluzek]

Thanks for working on this @mvertens and @billsacks -- yes this covers what we thought was important. This will also make @slevis-lmwg process easier to bring this part in.

[ekluzek]

One thing I will point out is that the changes as now implemented do require a CMEPS and CISM update. The logical ctl%rof_from_glc won't work correctly to use an older CISM version, because it's set too late. I'll make an issue regarding this and I'll assume that will be fixed in a later PR. It sounds like there is a plan for some follow on PR's as well.

[ekluzek]

OK I made an issue in #103 about the point I made just above.

[slevis-lmwg]

Made a new tag:

git tag -a mosart1.1.02
git push escomp mosart1.1.02