Update for HAFS coupling mode and changes that are required for WW3 coupling under UFS

[uturuncoglu]

Description of changes

This PR aims to bring HAFS related following changes to the upstream,

• The HAFS specific coupling mode is updated by removing distinction between the HAFS coupled modeling configurations with has active atmosphere (FV3) vs. data atmosphere (ERA5). Now, both configuration uses same set of exchange fields. The system_typeoption is removed.
• The exchange fields that is used in the HAFS coupling mode is now depend on activated model components. This was required to adapt recent changes in the FV3 cap.
• The couple of place in the code is fixed because it was failing when there is no exchange field in the specified coupling direction. This was required to allow atmosphere-wave coupling configuration with uni-directional interaction between them (i.e. atm->wav or wav->atm).
• The post phase for wave coupling is added
• Since NOAA/EMC WW3 cap is running in R4 mode by default, R4 support is added to med_methods_Grid_Print() call
• The wav->atm direction is added to allowed coupling directions

Specific notes

Contributors other than yourself, if any: @danrosen25

CMEPS Issues Fixed (include github issue #):

Are changes expected to change answers?

• [x] bit for bit
• [ ] different at roundoff level
• [ ] more substantial

Any User Interface Changes (namelist or namelist defaults changes)?

• [ ] Yes
• [x] No

Testing performed if application target is CESM:(either UFS-S2S or CESM testing is required):

• [ ] (recommended) CIME_DRIVER=nuopc scripts_regression_tests.py

  • machines: Cheyenne, intel/19.1.1 and mpt/2.22
  • details (e.g. failed tests): will be done along with CDEPS PR (https://github.com/ESCOMP/CDEPS/pull/107)

• [ ] (recommended) CESM testlist_drv.xml

  • machines and compilers:
  • details (e.g. failed tests):

• [x] (optional) CESM prealpha test

  • machines and compilers: Cheyenne, intel/19.1.1 and mpt/2.22
  • details (e.g. failed tests): Following tests are failed but we know that these tests fails also in the ESMF new alarm implementation.

    DAE_N2_D_Lh12_Vnuopc.f10_f10_mg37.I2000Clm50BgcCrop.cheyenne_intel.clm-DA_multidrv (Overall: FAIL) details:
    FAIL DAE_N2_D_Lh12_Vnuopc.f10_f10_mg37.I2000Clm50BgcCrop.cheyenne_intel.clm-DA_multidrv RUN time=1223
    ERC_D_Ln9_Vnuopc.mpasa480z32_mpasa480.FHS94.cheyenne_intel.cam-outfrq3s_usecase (Overall: FAIL) details:
    FAIL ERC_D_Ln9_Vnuopc.mpasa480z32_mpasa480.FHS94.cheyenne_intel.cam-outfrq3s_usecase MODEL_BUILD time=15
    ERP_D_Ln9_Vnuopc.C48_C48_mg17.QPC6.cheyenne_intel.cam-outfrq9s (Overall: FAIL) details:
    FAIL ERP_D_Ln9_Vnuopc.C48_C48_mg17.QPC6.cheyenne_intel.cam-outfrq9s MODEL_BUILD time=105
    ERP_D_Ln9_Vnuopc.f09_f09_mg17.FSD.cheyenne_intel.cam-outfrq9s_contrail (Overall: FAIL) details:
    FAIL ERP_D_Ln9_Vnuopc.f09_f09_mg17.FSD.cheyenne_intel.cam-outfrq9s_contrail RUN time=180
    IRT_N3_PM3_Ld7_Vnuopc.f19_g17.BHIST.cheyenne_intel.allactive-defaultio (Overall: FAIL) details:
    FAIL IRT_N3_PM3_Ld7_Vnuopc.f19_g17.BHIST.cheyenne_intel.allactive-defaultio SHAREDLIB_BUILD time=9
    NCK_Ld5_Vnuopc.f19_g17.B1850G.cheyenne_intel.allactive-cism-test_coupling (Overall: FAIL) details:
    FAIL NCK_Ld5_Vnuopc.f19_g17.B1850G.cheyenne_intel.allactive-cism-test_coupling COMPARE_base_multiinst
    NCK_Vnuopc.f19_g17.B1850.cheyenne_intel.allactive-defaultiomi (Overall: FAIL) details:
    FAIL NCK_Vnuopc.f19_g17.B1850.cheyenne_intel.allactive-defaultiomi COMPARE_base_multiinst
    SMS_Lm13_Vnuopc.f10_f10_mg37.I1850Clm50SpG.cheyenne_intel (Overall: FAIL) details:
    FAIL SMS_Lm13_Vnuopc.f10_f10_mg37.I1850Clm50SpG.cheyenne_intel RUN time=1
    SMS_Ln9_Vnuopc.f19_f19_mg17.FXHIST.cheyenne_intel.cam-outfrq9s_multi (Overall: FAIL) details:
    FAIL SMS_Ln9_Vnuopc.f19_f19_mg17.FXHIST.cheyenne_intel.cam-outfrq9s_multi RUN time=1822
    SMS_Vnuopc_Lm13.f10_f10_mg37.I1850Clm50SpG.cheyenne_intel (Overall: FAIL) details:
    FAIL SMS_Vnuopc_Lm13.f10_f10_mg37.I1850Clm50SpG.cheyenne_intel RUN time=1

    Here is the detail investigation of the errors,

• _DAE_N2_D_Lh12_Vnuopc.f10_f10mg37.I2000Clm50BgcCrop: It runs but killed by the scheduler due to time out.

• _ERC_D_Ln9_Vnuopc.mpasa480z32mpasa480.FHS94: fails in build with following error. /glade/work/turuncu/TESTING/CESM_hafs/components/cam/src/dynamics/mpas/dycore/src/framework/mpas_derived_types.F(35): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [ESMF_STUBS]

• _ERP_D_Ln9_Vnuopc.C48_C48mg17.QPC6: I checkout externals by providing -o option as @jedwards4b suggested and I have ../../libraries/FMS/ directory now but the test still fails with /glade/work/turuncu/TESTING/CESM_hafs/components/cam/src/dynamics/fv3/atmos_cubed_sphere/tools/fv_mp_mod.F90(75): error #6580: Name in only-list does not exist or is not accessible. [MPP_NODE] error.

• _ERP_D_Ln9_Vnuopc.f09_f09_mg17.FSD.cheyenne_intel.cam-outfrq9scontrail: It gives floating point error in atm PEs.

• _IRT_N3_PM3_Ld7_Vnuopc.f19g17.BHIST: It could not find suitable decomposition configuration and fails with CAM build-namelist - ERROR: fv_decomp_set failed to find a decomposition. error.

• _NCK_Ld5_Vnuopc.f19_g17.B1850G and NCK_Vnuopc.f19g17.B1850: It runs but there are Andover changes and do not validate.

• _SMS_Lm13_Vnuopc.f10_f10mg37.I1850Clm50SpG: It dies with BlockingIOError: [Errno 11] Resource temporarily unavailable: during run.

• _SMS_Ln9_Vnuopc.f19_f19mg17.FXHIST: It runs but killed by the scheduler due to time out.

• _SMS_Vnuopc_Lm13.f10_f10mg37.I1850Clm50SpG Again, it dies with BlockingIOError: [Errno 11] Resource temporarily unavailable: during run.

• [ ] (other) please described in detail

  • machines and compilers
  • details (e.g. failed tests):

Testing performed if application target is UFS-coupled:

• [ ] (recommended) UFS-coupled testing
  • description:
  • details (e.g. failed tests):

Testing performed if application target is UFS-HAFS:

• [x] (recommended) UFS-HAFS testing
  • description: The HAFS application is extensively tested against different configurations including data atmosphere, data ocean and fully coupled three component (atm-ocn-wav) cases.
  • details (e.g. failed tests): None

Hashes used for testing:

• [x] CESM:
  • repository to check out: https://github.com/ESCOMP/CESM.git
  • branch:
  • hash: cesm2_3_beta04
• [x] UFS-coupled, then umbrella repostiory to check out and associated hash:
  • repository to check out: https://github.com/hafs-community/ufs-weather-model
  • branch:
  • hash: support/HAFS
• [x] UFS-HAFS, then umbrella repostiory to check out and associated hash:
  • repository to check out: https://github.com/ESCOMP/CESM.git
  • branch:
  • hash: cesm2_3_beta04

[uturuncoglu]

All CESM prealpha test are run and the failed tests are listed in the PR description. It seems that they are not related with this PR. I'll also run CESM testlist_drv.xml and nuopc scripts_regression_tests.py

[mvertens]

@uturuncoglu - you do not need to run the testlist_drv.xml if you have run the prealpha tests.

[uturuncoglu]

@mvertens Okay. Actually, I have already run them but I could not access to the results because Cheyenne is down.

[uturuncoglu]

@jedwards4b @DeniseWorthen I just wonder if you find chance to look at this PR.

[DeniseWorthen]

@uturuncoglu I'm sorry, I forgot. Have you tested this PR in the ufs-weather-model RT? It wasn't quite clear to me in the description.

[uturuncoglu]

@DeniseWorthen Yes, this PR is tested couple of times under UFS by running full RTs as well as running HAFS specific RTs (they are not part of UFS yet) and HAFS application tests and there is no issue. Here is the specific branch that is tested

https://github.com/hafs-community/ufs-weather-model/tree/feature/hafs_sync_20210716

it has both CDEPS and WW3 related changes. Maybe you could just run the standard coupled case in your side to double check and make sure.

[DeniseWorthen]

@uturuncoglu That's great. I can approve now.

[uturuncoglu]

@jedwards4b I run the nuopc scripts_regression_tests.py but I don't have the results since Cheyenne is down. CESM prealpha tests were fine. If you want I could wait for merge until Cheyenne will be available again.