Update ufs-bundle to build with head of develop of ufs-weather-model for ATM configuration

[mark-a-potts]

Description

This PR, in combination with PRs for oops and fv3-jedi, allows the ufs-bundle to be built using the head of develop for the ufs-weather-model. Additionally, new regression tests have been added to fv3-jedi that will update the run directories needed for the UFS model and to run a sample Ensemble Forecast of the UFS.

Issue(s) addressed

Resolves #14

Dependencies

List the other PRs that this PR is dependent on:

• [x] https://github.com/JCSDA-internal/fv3-jedi/pull/1099

Impact

None.

Checklist

• [X] I have performed a self-review of my own code
• [x] I have made corresponding changes to the documentation
• [X] I have run the unit tests before creating the PR

[mark-a-potts]

This PR has been tested and works with UFS_APP=ATM, ATMAERO, and S2S configurations. There are no tests for ATMAERO or S2S, but both configurations build. The NG-GODAS option does not build because the soca branch required is out of date with the rest of the bundle and requires fairly significant work.

[climbfuji]

This PR has been tested and works with UFS_APP=ATM, ATMAERO, and S2S configurations. There are no tests for ATMAERO or S2S, but both configurations build. The NG-GODAS option does not build because the soca branch required is out of date with the rest of the bundle and requires fairly significant work.

I have an updated SOCA branch that I used for a similar effort ... I can make that work with this PR.

[mark-a-potts]

That would be great. I started on a merge and it looked like it was going to be a lot of work. I added a patch to handle an issue with checksum checking of restart files down in atmos_cubed_sphere. I know that there are differences in the MOM6 repo, so maybe we could add a similar patch for those and get everything working with the head of develop of the WM.

[climbfuji]

@mark-a-potts Let's fix the CI failures first. This is because of a merge last week on the JEDI side that changed the model interfaces, but nobody thought about the fv3-jedi etc branches for the UFS. Let me do this in an hour or so, ok?

[mark-a-potts]

I have updated this to now handle building ATM, ATMAERO, and S2S using the head of develop of the ufs-wm. NG-GODAS is still very broken, but the build with S2S will actually run the ATM regression tests without issue.

[climbfuji]

@mark-a-potts I resolved the merge conflict in https://github.com/JCSDA/ufs-bundle/pull/43/commits/dd72818159051e4e0958ef0d9a6ce9df78b1ff43, please take a look to see if everything looks ok

[climbfuji]

There is now a CI failure in the cmake step, related to fv3-jedi-data: https://github.com/JCSDA/ufs-bundle/actions/runs/7133595917/job/19426615092?pr=43

[mark-a-potts]

Just removed the extra fv3-jedi-data section. I think it should work now.

[climbfuji]

Sigh, there are so many updates to the JEDI code that it seems impossible to get a clean run! I'll hop on the instance and clear the build area, then restart the test.

[mark-a-potts]

The latest update to ioda seems to have broken the buid now.

[climbfuji]

The latest update to ioda seems to have broken the buid now.

Can you try to pull oops develop into your branch please?

[mark-a-potts]

Yeah, I think that was the issue. Just re-built successfully on my machine, so pushing now.

[climbfuji]

@mark-a-potts I am also running into problems when having to rerun make. Whether UPDATE_DISCONNECTED is ON or OFF, I get something like this:

[  0%] Performing update step for 'ufs-weather-model'
cd /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ensforecasts/ufs-weather-model && /opt/homebrew/Cellar/cmake/3.27.0/bin/cmake -Dcan_fetch=YES -P /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ensforecasts/build-atm/ufs-weather-model/tmp/ufs-weather-model-gitupdate.cmake
No stash entries found.
No stash entries found.
No stash entries found.
CMake Error at /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ensforecasts/build-atm/ufs-weather-model/tmp/ufs-weather-model-gitupdate.cmake:277 (message):

  Failed to unstash changes in:
  '/Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ensforecasts/ufs-weather-model'.

  You will have to resolve the conflicts manually

make[2]: *** [ufs-weather-model/src/ufs-weather-model-stamp/ufs-weather-model-update] Error 1
make[1]: *** [CMakeFiles/ufs-weather-model.dir/all] Error 2
make: *** [all] Error 2

This may have to do with the patch that gets applied to the ufs-weather-model develop branch?

[climbfuji]

@mark-a-potts I am also running into problems when having to rerun make. Whether UPDATE_DISCONNECTED is ON or OFF, I get something like this:

[  0%] Performing update step for 'ufs-weather-model'
cd /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ensforecasts/ufs-weather-model && /opt/homebrew/Cellar/cmake/3.27.0/bin/cmake -Dcan_fetch=YES -P /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ensforecasts/build-atm/ufs-weather-model/tmp/ufs-weather-model-gitupdate.cmake
No stash entries found.
No stash entries found.
No stash entries found.
CMake Error at /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ensforecasts/build-atm/ufs-weather-model/tmp/ufs-weather-model-gitupdate.cmake:277 (message):

  Failed to unstash changes in:
  '/Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ensforecasts/ufs-weather-model'.

  You will have to resolve the conflicts manually

make[2]: *** [ufs-weather-model/src/ufs-weather-model-stamp/ufs-weather-model-update] Error 1
make[1]: *** [CMakeFiles/ufs-weather-model.dir/all] Error 2
make: *** [all] Error 2

This may have to do with the patch that gets applied to the ufs-weather-model develop branch?

Update. I did a clean checkout and ran into the same problem:

[  0%] Performing disconnected update step for 'ufs-weather-model'
cd /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ensforecasts/ufs-weather-model && /opt/homebrew/Cellar/cmake/3.27.0/bin/cmake -Dcan_fetch=NO -P /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ensforecasts/build-atm/ufs-weather-model/tmp/ufs-weather-model-gitupdate.cmake
No stash entries found.
No stash entries found.
No stash entries found.
CMake Error at /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ensforecasts/build-atm/ufs-weather-model/tmp/ufs-weather-model-gitupdate.cmake:277 (message):

  Failed to unstash changes in:
  '/Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ensforecasts/ufs-weather-model'.

  You will have to resolve the conflicts manually

make[2]: *** [ufs-weather-model/src/ufs-weather-model-stamp/ufs-weather-model-update_disconnected] Error 1
make[1]: *** [CMakeFiles/ufs-weather-model.dir/all] Error 2
make: *** [all] Error 2

Since the patch only applies to the top-level ufs-weather-model and not to any of the submodules underneath, I would say we use a branch for the moment. The ufs-weather-model developers have indicated that they are open to these changes, therefore I expect that we wouldn't have to maintain that ufs-weather-model branch for very long.

[mark-a-potts]

I cleaned up the CMakeLists.txt file a bit and fixed the *-data bundle syntax to match the other bundles. Note that the ufs-weather-model is now pointing to my PR branch (feature/fortran-flags) in github.com/NOAA-EPIC/ufs-weather-model. I got rid of the mom6 patch and added an ng-godas patch instead. It gets run at the top level of ufs-weather-model, but it is actually patching the MOM6 repo under MOM6-interface. I have successfully built using UFS_APP=NG-GODAS and S2S, but still waiting on ATM and ATMAERO.

[climbfuji]

I cleaned up the CMakeLists.txt file a bit and fixed the *-data bundle syntax to match the other bundles. Note that the ufs-weather-model is now pointing to my PR branch (feature/fortran-flags) in github.com/NOAA-EPIC/ufs-weather-model. I got rid of the mom6 patch and added an ng-godas patch instead. It gets run at the top level of ufs-weather-model, but it is actually patching the MOM6 repo under MOM6-interface. I have successfully built using UFS_APP=NG-GODAS and S2S, but still waiting on ATM and ATMAERO.

That sounds great, thanks very much @mark-a-potts

[climbfuji]

@mark-a-potts Looks like CI tests are passing now !

[mark-a-potts]

They didn't actually all run @climbfuji But, it is close now. I will try submitting again.

[mark-a-potts]

This time the tests actually ran and seem to have passed. Yay!

[climbfuji]

@mark-a-potts I am unable to build any of the APPs with this PR. NO matter what I do, I get this from the make command after a fresh checkout and fresh cmake run:

[  0%] Performing disconnected update step for 'ufs-weather-model'
cd /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ufs-ensforcasts/ufs-weather-model && /opt/homebrew/Cellar/cmake/3.27.0/bin/cmake -Dcan_fetch=NO -P /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ufs-ensforcasts/build-atm/ufs-weather-model/tmp/ufs-weather-model-gitupdate.cmake
No stash entries found.
No stash entries found.
No stash entries found.
CMake Error at /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ufs-ensforcasts/build-atm/ufs-weather-model/tmp/ufs-weather-model-gitupdate.cmake:277 (message):

  Failed to unstash changes in:
  '/Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ufs-ensforcasts/ufs-weather-model'.

  You will have to resolve the conflicts manually

make[2]: *** [ufs-weather-model/src/ufs-weather-model-stamp/ufs-weather-model-update_disconnected] Error 1
make[1]: *** [CMakeFiles/ufs-weather-model.dir/all] Error 2
make: *** [all] Error 2

[mark-a-potts]

@mark-a-potts I am unable to build any of the APPs with this PR. NO matter what I do, I get this from the make command after a fresh checkout and fresh cmake run:

[  0%] Performing disconnected update step for 'ufs-weather-model'
cd /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ufs-ensforcasts/ufs-weather-model && /opt/homebrew/Cellar/cmake/3.27.0/bin/cmake -Dcan_fetch=NO -P /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ufs-ensforcasts/build-atm/ufs-weather-model/tmp/ufs-weather-model-gitupdate.cmake
No stash entries found.
No stash entries found.
No stash entries found.
CMake Error at /Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ufs-ensforcasts/build-atm/ufs-weather-model/tmp/ufs-weather-model-gitupdate.cmake:277 (message):

  Failed to unstash changes in:
  '/Users/heinzell/work/ufs-bundle/ufs-bundle-mark-ufs-ensforcasts/ufs-weather-model'.

  You will have to resolve the conflicts manually

make[2]: *** [ufs-weather-model/src/ufs-weather-model-stamp/ufs-weather-model-update_disconnected] Error 1
make[1]: *** [CMakeFiles/ufs-weather-model.dir/all] Error 2
make: *** [all] Error 2

So, this might be because you need to remove the old ufs-weather-model directory? To accommodate the addition of the fortran flags for building, I switched to a fork of the WM under NOAA-EPIC, so if it still thinks it is looking for the branch under ufs-community, it will fail.

[mark-a-potts]

I just checked with @jkbk2004, and it sounds like my PR to the UFS-WM should go in later this week, but in the meantime, I extended the patch and will push up a new CMakeLists.txt file that pulls from ufs-community instead of NOAA-EPIC.

[mark-a-potts]

If it is exclusively a case-sensitivity issue, this might be an alternative for your testing--

https://brianboyko.medium.com/a-case-sensitive-src-folder-for-mac-programmers-176cc82a3830

We should be able to remove the patch to fix the CMakeLists.txt files when my PR into the WM gets merged, but the rest of that patch fixes issues in MOM6 that we need for coupling to SOCA. Given that the NG-GODAS coupling is broken right now anyway, I would probably lean towards removing the NG-GODAS option from ufs-bundle for the time being.

We could also put in a PR to MOM6, but I feel kind of uncomfortable doing that if the SOCA connection doesn't really work right now.

[mark-a-potts]

The oops PR is going to take a while, but I think I can comment out the ensemble test in fv3-jedi and get this working. I'll try that today.

[climbfuji]

Last thing is to get the fv3-jedi PR merged, correct? I think you avoided the oops dependency (it's still in the PR description) for now.

[mark-a-potts]

Yes. The fv3-jedi PR can be done without the oops PR, and I think it would be good to do that. I will remove the mentions of oops in the PR descriptions.

[climbfuji]

There is also a conflict for README.md because my last PR replaced w3nco with w3emc.

[mark-a-potts]

Working on this now. The s3 bucket is currently being synched to the latest rt results, so that is delaying the testing a bit. Will update shortly.

[mark-a-potts]

After the latest update, the ctests pass and builds now work for ATM, ATMAERO, NG-GODAS, and S2S configurations.

[climbfuji]

@mark-a-potts CI tests are failing unfortunately: https://github.com/JCSDA/ufs-bundle/actions/runs/7745452671/job/21121475447?pr=43