Accommodate multi-instance runs in CESM

[alperaltuntas]

This PR introduces changes that are mainly in NUOPC cap, infra/FMS, and src/framework to accommodate multi-instance runs in CESM:

Changes in nuopc cap:

• receive instance id and suffix from the CESM coupler.
• call ensemble_manager_init. This makes sure the input/output files contain the instance suffix.
• add instance suffix to log files, restart files, and rpointer.

Changes in infra/FMS{1,2}:

• add the optional ensumble_suffix arg to ensemble_manager_init. If provided, this arg bypasses the call to FMS_ensemble_manager_init, and instead, calls fms_io_set_filename_appendix and fms2_io_set_filename_appendix. This makes sure that all i/o files managed my FMS contains the ensemble suffix.

Change in src/framework:

• append instance suffix to MOM_parameter_doc* files. ( I believe this is the only change that may potentially impact GFDL's ensemble workflow.)

This PR should be evaluating in conjunction with https://github.com/ESCOMP/MOM_interface/pull/137.

I am mentioning @MJHarrison-GFDL, @marshallward, and @hkershaw-brown in case they'd prefer to review this PR before it gets merged to dev/ncar. If so, please let us know and will hold off until you review.

[codecov-commenter]

Codecov Report

Patch coverage: 31.49% and project coverage change: +1.30 :tada:

Comparison is base (59cc9ed) 35.88% compared to head (7344fcd) 37.18%.

:exclamation: Current head 7344fcd differs from pull request most recent head fb10ddc. Consider uploading reports for the commit fb10ddc to get more accurate results

Additional details and impacted files

```diff @@ Coverage Diff @@ ## dev/ncar #241 +/- ## ============================================ + Coverage 35.88% 37.18% +1.30% ============================================ Files 260 263 +3 Lines 72088 73207 +1119 Branches 13601 13621 +20 ============================================ + Hits 25868 27223 +1355 + Misses 41171 40976 -195 + Partials 5049 5008 -41 ``` | [Impacted Files](https://codecov.io/gh/NCAR/MOM6/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR) | Coverage Δ | | |---|---|---| | [src/core/MOM\_unit\_tests.F90](https://codecov.io/gh/NCAR/MOM6/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR#diff-c3JjL2NvcmUvTU9NX3VuaXRfdGVzdHMuRjkw) | `16.66% <ø> (ø)` | | | [src/framework/MOM\_get\_input.F90](https://codecov.io/gh/NCAR/MOM6/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR#diff-c3JjL2ZyYW1ld29yay9NT01fZ2V0X2lucHV0LkY5MA==) | `62.16% <0.00%> (ø)` | | | [src/tracer/MOM\_neutral\_diffusion.F90](https://codecov.io/gh/NCAR/MOM6/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR#diff-c3JjL3RyYWNlci9NT01fbmV1dHJhbF9kaWZmdXNpb24uRjkw) | `65.38% <ø> (ø)` | | | [src/tracer/MOM\_tracer\_types.F90](https://codecov.io/gh/NCAR/MOM6/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR#diff-c3JjL3RyYWNlci9NT01fdHJhY2VyX3R5cGVzLkY5MA==) | `0.00% <ø> (ø)` | | | [src/framework/MOM\_file\_parser.F90](https://codecov.io/gh/NCAR/MOM6/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR#diff-c3JjL2ZyYW1ld29yay9NT01fZmlsZV9wYXJzZXIuRjkw) | `90.45% <25.00%> (+30.25%)` | :arrow_up: | | [src/tracer/MOM\_tracer\_hor\_diff.F90](https://codecov.io/gh/NCAR/MOM6/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR#diff-c3JjL3RyYWNlci9NT01fdHJhY2VyX2hvcl9kaWZmLkY5MA==) | `76.51% <25.00%> (+0.03%)` | :arrow_up: | | [src/tracer/MOM\_hor\_bnd\_diffusion.F90](https://codecov.io/gh/NCAR/MOM6/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR#diff-c3JjL3RyYWNlci9NT01faG9yX2JuZF9kaWZmdXNpb24uRjkw) | `61.87% <29.41%> (ø)` | | | [src/tracer/MOM\_tracer\_registry.F90](https://codecov.io/gh/NCAR/MOM6/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR#diff-c3JjL3RyYWNlci9NT01fdHJhY2VyX3JlZ2lzdHJ5LkY5MA==) | `64.72% <33.33%> (+0.18%)` | :arrow_up: | | [...nfig\_src/infra/FMS1/MOM\_ensemble\_manager\_infra.F90](https://codecov.io/gh/NCAR/MOM6/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR#diff-Y29uZmlnX3NyYy9pbmZyYS9GTVMxL01PTV9lbnNlbWJsZV9tYW5hZ2VyX2luZnJhLkY5MA==) | `30.00% <75.00%> (+2.22%)` | :arrow_up: | | [src/core/MOM.F90](https://codecov.io/gh/NCAR/MOM6/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR#diff-c3JjL2NvcmUvTU9NLkY5MA==) | `51.18% <100.00%> (ø)` | | | ... and [1 more](https://codecov.io/gh/NCAR/MOM6/pull/241?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR) | | ... and [9 files with indirect coverage changes](https://codecov.io/gh/NCAR/MOM6/pull/241/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR) Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAR)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[marshallward]

The changes to MOM and the FMS infra should be fine. I have a pending PR which removes the references to fms_io_mod in the FMS2 ensemble manager, but that conflict should not be difficult to resolve when it happens.

(I would suggest looking into squashing and rebasing some of these commits though!)

[hkershaw-brown]

I ran an 8 instance case with this, looks good.

[alperaltuntas]

@gustavo-marques , I just realized there is a conflict due to the latest PR merge. I'll address that.

[alperaltuntas]

@gustavo-marques this is ready to merge. You can squash and merge. To run with multiple instances you can set ./xmlchange NINST_OCN=N where N is the number of MOM6 instances you'd like to run with.