refix forcing groups

[mark-petersen]

This change is required to run with multiple forcing groups in ACME, like both salinity restoring and BGC. They were put in by Adrian in PR #1333, but then removed inadvertently in PR #1427.

[mark-petersen]

@jonbob, @vanroekel and @maltrud, I'm out this week. I ran all my standard tests in MPAS-Ocean on this and it passed. If it works in ACME, any of you can go ahead and merge it.

[vanroekel]

@jonbob the changes look good to me visually. Could you run the domain decomposition and threading tests for salinity restoring to verify no unforeseen issues arise in E3SM?

[maltrud]

@mark-petersen, @vanroekel, @jonbob i tested Mark's branch using GMPAS-OIECO-OIDMS-IAF_oEC60to30v3 with both eco forcing and SSS restoring and it appears to work fine. the restart files have both of the correct forcing group names, and it also restarted without a problem. i didn't check exact restart but hope to do that fairly soon. but i'd say don't wait for that.

[jonbob]

Testing within E3SM:

• passes ERS_Ld5_P480.T62_oEC60to30v3.GMPAS-OIECO-IAF.edison_gnu w/ BGC only
• passes ERS_Ld5_P480.T62_oEC60to30v3.GMPAS-OIECO-IAF.edison_gnu w/ BGC and salinity restoring
• passes PEM_Ld5_P480.T62_oEC60to30v3.GMPAS-OIECO-IAF.edison_intel w/BGC only

[jonbob]

Continued testing:

• fails PET_Ld5_P480.T62_oEC60to30v3.GMPAS-OIECO-IAF.edison_intel w/ BGC and salinity restoring

I'll try testing with just BGC and then also with just salinity restoring

[jonbob]

Continued testing:

• fails PET_Ld5_P480.T62_oEC60to30v3.GMPAS-OIECO-IAF.edison_intel w/ BGC

[jonbob]

Continued testing:

• passes PET_Ld5_P480.T62_oEC60to30v3.GMPAS-IAF.edison_intel (salinity restoring only)

[jonbob]

Obviously a problem with threading and BGC. Do we want to continue with this PR and open an issue for this? Or put this PR on hold and fix here? @maltrud ? @mark-petersen ?

[mark-petersen]

I'd like to merge this as-is because it adds back some previously removed lines that are needed and pass the ERS test with both salinity restoring and BGC. Then let's look at the BGC threading problem separately. @jonbob or @maltrud if you agree, go ahead and approve this PR.

[jonbob]

@mark-petersen - as soon as you want to get this into mpas-dev, we can make an ACME PR as well

[maltrud]

i agree. i thought i had caught all of the threading issues with BGC, but looks like not.