[production/AQM.v7] Adding a capability for handling the cases of missing restart files to support AQMv7 IT test

[JianpingHuang-NOAA]

DESCRIPTION OF CHANGES:

1. Modify make_ics ex-script for handling missing previous cycle restart files. If two previous cycles restart files are missing, then use the climatology restart file
2. Create tracer restart file and coupler.res based on the case on 20231007 and put them under ~/fix/restart
3. Add one more dependency of make_ics for aqm_lbcs.sh
4. Further refine wcoss2.yale file

• [ ] hera.intel
• [ ] orion.intel
• [ ] hercules.intel
• [ ] cheyenne.intel
• [ ] cheyenne.gnu
• [ ] derecho.intel
• [ ] gaea.intel
• [ ] gaeac5.intel
• [ ] jet.intel
• [x] wcoss2.intel
• [ ] NOAA Cloud (indicate which platform)
• [ ] Jenkins
• [ ] fundamental test suite
• [ ] comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

This PR is used to fix issue 926

CHECKLIST

• [x] My code follows the style guidelines in the Contributor's Guide
• [x ] I have performed a self-review of my own code using the Code Reviewer's Guide
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] My changes need updates to the documentation. I have made corresponding changes to the documentation
• [ ] My changes do not require updates to the documentation (explain).
• [ ] My changes generate no new warnings
• [ ] New and existing tests pass with my changes
• [ ] Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• [ ] Work In Progress
• [ ] bug
• [x] enhancement
• [ ] documentation
• [ ] release
• [x] high priority
• [ ] run_ci
• [ ] run_we2e_fundamental_tests
• [ ] run_we2e_comprehensive_tests
• [ ] Needs Cheyenne test
• [ ] Needs Jet test
• [ ] Needs Hera test
• [ ] Needs Orion test
• [ ] help wanted

CONTRIBUTORS (optional):

By Kai Wang and Jianping Huang (EMC)

[JianpingHuang-NOAA]

@chan-hoo Please link /lfs/h2/emc/physics/noscrub/UFS_SRW_App/aqm.v7/fix to your location under the package in a parallel position of ush after the code compilation is completed and before you launch the workflow for a test.

[chan-hoo]

@JianpingHuang-NOAA, my test run failed. I think the above link caused this issue. Can you add the above link command to your PR?

[JianpingHuang-NOAA]

@chan-hoo I tried to modify the "generate_FV3LAM_wflow.py" to create symbolic link for AQM automatically for community user. It was ok for all other links except for ~/fix/fix_lam (see /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.85f/ush/generate_FV3LAM_wflow.py_new_failed).

, then I tried to do that through modifying the "link_fix.py" but somehow the values of Environmental Variables $HOMEaqm or $FIXaqm are not passed into the link_fix.py. This requires your support on this. If this is time consumed, please link the fix directory from "/lfs/h2/emc/physics/noscrub/UFS_SRW_App/aqm.v7/fix" to your package manually. This is the way we proposed to NCO for implementation. Please following this way to complete this test and get the PR merged so that we can move the project forward. Thanks ! @HaixiaLiu-NOAA

[HaixiaLiu-NOAA]

Based on the comment above "please link the fix directory from "/lfs/h2/emc/physics/noscrub/UFS_SRW_App/aqm.v7/fix" to your package manually. This is the way we proposed to NCO for implementation", if this is the NCO approved method to implement the AQMv7 workflow, @chan-hoo please proceed with making the link and progressing with this PR. @JianpingHuang-NOAA please confirm that is the correct process @chan-hoo to go with.

[chan-hoo]

@JianpingHuang-NOAA, @HaixiaLiu-NOAA, I'll try to find a (temporary) solution and let you know soon.

[chan-hoo]

@JianpingHuang-NOAA, can you replace your ush/setup.py file with the following?

/lfs/h2/emc/lam/noscrub/chan-hoo.jeon/prod_aqm_test/ufs-srweather-app/ush/setup.py

This change looks working well for the fix directory issue. My test run failed in some bias-correction tasks due to missing data files, but I don't want to talk about this any more in this PR. However, I guess it may work well on your account. Please let me know if it (linking fix dir) is what you want.

[chan-hoo]

@JianpingHuang-NOAA, so does this work for you too?

[JianpingHuang-NOAA]

@chan-hoo Yes, it works for 00z cycle on 20231016. I am testing 06z cycle for a double check.

Please see all the results at /lfs/h2/emc/ptmp/jianping.huang/emc.para/com/aqm/v7.0/c85f.20231016/00 (Dogwood).

[chan-hoo]

@JianpingHuang-NOAA, for a next PR, I'd like to merge this PR now. What do you think?

[JianpingHuang-NOAA]

@chan-hoo 06z cycles is running as we expected. I did not see any issue. Please go ahead to merge it. Thanks !