search

ESMCI / git-fleximod

MIT License
2 stars 2 forks source link

How to check out updated nested submodules if changing top-level submodule commit #50

Open jimmielin opened 4 days ago

jimmielin commented 4 days ago

Hello,

Disclaimers: (1) Sorry for the possibly incomprehensible title. (2) I understand development code is unsupported.

Background

@lizziel and I are developing an update to GEOS-Chem chemistry within CAM. The new update has both new submodules under CAM, and updates to old submodules. Something like this:

Old

ESCOMP/CESM @ cesm3_0_beta01
-- components/cam: ESCOMP/CAM @ cam6_3_162
---- components/cam/src/chemistry/geoschem/geoschem_src: geoschem/geos-chem @ 14.1.2
---- components/cam/src/hemco: ESCOMP/HEMCO @ hemco-cesm1_2_1_hemco3_6_3_cesm_rmebranch

New

ESCOMP/CESM @ cesm3_0_beta01
-- components/cam: geoschem/CAM @ feature/geoschem_14.4
---- components/cam/src/chemistry/geoschem/geoschem_src: geoschem/geos-chem @ feature/geoschem_14.4_in_cesm
---- components/cam/src/hemco: ESCOMP/HEMCO @ dev/hemco-cesm1_4_0_hemco3_9_0
---- components/cam/src/chemistry/hetp: ** new external **

What has happened?

I want to checkout ESCOMP/CESM at cesm3_0_beta01, and update to use our fork of CAM.

Analogous to usage of manage_externals, I change .gitmodules section on cam to be:

[submodule "cam"]
        path = components/cam
        url = https://www.github.com/geoschem/CAM
        fxDONOTUSEurl = https://www.github.com/ESCOMP/CAM
        fxtag = feature/geoschem_14.4
        fxrequired = ToplevelRequired

Then run ./bin/git-fleximod update.

The expected result is that components/cam is checked out at feature/geoschem_14.4, and the externals in CAM, i.e., geoschem is checked out at feature/geoschem_14.4_in_cesm, hemco is checked out at dev/..., and the new external hetp is checked out.

What actually happens is that feature/geoschem_14.4 is checked out in components/cam, but the underlying submodules use what's specified in cam6_3_162:

Recursively checking out submodules of cam

...
e             geoschem not checked out, aligned at tag 14.1.2
e                hemco not checked out, aligned at tag hemco-cesm1_2_1_hemco3_6_3_cesm_rme
...

                 cam updated to feature/geoschem_14.4

If components/cam is navigated you can see that git status shows the geoschem and hemco submodules have been checked out at their cam6_3_162 commits and not the ones specified in feature/geoschem_14.4.

Why?

The problem is that Git doesn't actually store the submodule commit hash in .gitmodules, unlike how it is stored by manage_externals in Externals.cfg. This is, of course, standard Git behavior, but a manage_externals user might not realize this, and the presence of fxtag in .gitmodules could imply that git-fleximod will help replicate old behavior.

The correct way to checkout an updated CAM external after checking out ESCOMP/CESM, is to update the Git index:

git update-index --cacheinfo 160000,a126198071fe4e831cd1465d44e6eb0dd3f9cb1a,components/cam

To point components/cam to the commit corresponding to feature/geoschem_14.4; then, .gitmodules can be updated so fxtag is the same, then ./bin/git-fleximod update will work as expected.

Questions

What is the standard git-fleximod approach to switching a external to a different commit correctly (both in terms of convention and also handling the submodules within it correctly)?

Is git-fleximod considering to mimic the behavior of manage_externals by handling the case where fxtag specified in .gitmodules is out-of-sync with the git index?

Thanks!

jedwards4b commented 4 days ago

git-fleximod update only updates the sandbox - to update the submodule reference in the remote you need to add and commit the submodule - you can do this with git commit -a -m'update submodule'

jimmielin commented 4 days ago

Thanks @jedwards4b, what if I haven't checked out components/cam yet in the sandbox and I want to update the reference so it checks out a fork instead (which contains updated externals)?

Even after I git-fleximod update, go into components/cam and checkout feature/geoschem_14.4, whether or not I commit the updated ref into the sandbox will not change the behavior of git-fleximod update; it seems to ignore any new/updated externals under components/cam.

jimmielin commented 4 days ago

Minimal example:

git clone git@github.com:ESCOMP/CESM
cd CESM
sed -i "s/ESCOMP\/CAM/geoschem\/CAM/g" .gitmodules
sed -i "s/cam6_3_162/feature\/geoschem_14.4/g" .gitmodules
./bin/git-fleximod update

Expected: components/cam/src/chemistry/geoschem_src should be at feature/geoschem_14.4_in_cesm (commit 002a25d). The remote tree has the right submodule reference, see: https://github.com/geoschem/CAM/tree/feature/geoschem_14.4/src/chemistry/geoschem

Actual: components/cam/src/chemistry/geoschem_src is at 14.1.2, which is what's pointed from cam6_3_162: https://github.com/ESCOMP/CAM/tree/cam6_3_162/src/chemistry/geoschem

lizziel commented 4 days ago

I also wonder if this would work?

  1. clone cesm
  2. run fleximod in cesm clone
  3. cd components/cam
  4. add remote for geoschem/cam
  5. checkout cam branch
  6. run fleximod in cam

This is what you would do with the offline GEOS-Chem if the top-level did not have a commit of the submodule you want, except instead of running fleximod you do git submodule update --init --recursive.

lizziel commented 4 days ago

If that does work, why can't fleximod be run recursively in the submodules specified in .gitmodules?

jimmielin commented 4 days ago

Hi @lizziel, this doesn't work. I think it's similar to what my process is doing, git-fleximod update is checking out components/cam in the end and doesn't recurse to reinitialize the submodules.

What I'm doing:

cd components/cam
git remote add geoschem git@github.com:geoschem/CAM.git
git fetch geoschem
git checkout feature/geoschem_14.4

In accordance to git submodule behavior, the submodules are not updated at this stage:

➜  cam git:(ab476f9b) git checkout feature/geoschem_14.4
Updating files: 100% (25/25), done.
M       src/chemistry/geoschem/geoschem_src
M       src/hemco
Previous HEAD position was ab476f9b Merge pull request #1052 from jedwards4b/rme/cam6_3_161
branch 'feature/geoschem_14.4' set up to track 'geoschem/feature/geoschem_14.4'.
Switched to a new branch 'feature/geoschem_14.4'
➜  cam git:(feature/geoschem_14.4) ✗ git status
Refresh index: 100% (2198/2198), done.
git sOn branch feature/geoschem_14.4
Your branch is up to date with 'geoschem/feature/geoschem_14.4'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   src/chemistry/geoschem/geoschem_src (new commits)
        modified:   src/hemco (new commits)

no changes added to commit (use "git add" and/or "git commit -a")
u➜  cam git:(feature/geoschem_14.4) ✗ git submodule
-2ff978f92a5ac9a6ab243e5c14d06a7e2d2f5799 ccs_config
 f923081508f4264e61fcef2753a9898e55d1598e chem_proc (chem_proc5_0_06)
-422ddaa770a3cea6e83a60c9700ebce77acaceed cime
-7476950699909813d1938a34bd8d71bf5bfbf1e9 components/cdeps
-bdf6ea04d6133434fcaa4de5336de106f01290d0 components/cice
-c05dd5c4fc85327e76523aaea9cfe1e388748928 components/cism
-a9433779f0ae499d60ad118d2ec331628f0eaaa8 components/clm
-6384ff4e4a6bc82a678f9419a43ffbd5d53ac209 components/cmeps
-81c720c7ee51f9c69f2934f696078c42f4493565 components/mizuRoute
-8c682b1b7f15d146816de302e0d359da3e957056 components/mosart
-88503adbc275fb2ccbb6b598e460deaeb140e515 components/rtm
-270433531d33c64da7944d80564fe39a84917d26 libraries/FMS
-82b0071e69d14330b75d23b0bc68543ebea9aadc libraries/mct
-f52ade075619b32fa141993b5665b0fe099befc2 libraries/parallelio
-4b9dc4871a259f00f35bb47708d876cb7dcdf75c share
 4944547f04b1457d78bf7d3c0becddcfe0deabb9 src/atmos_phys (atmos_phys0_02_006)
-4bad0b7a722f70c01e7314aacf32eec5596ca068 src/chemistry/cloud_j
+28345ee76e5631d6d14869a36dc73e9dd6e0ce1e src/chemistry/geoschem/geoschem_src (preFlexChem-6085-g28345ee76)
-2a99b24625ed26cf87ae88697ddd6cf8bbdec812 src/chemistry/hetp
 df3550b0f6a835778f32ccc8c6291942e0413f62 src/dynamics/fv3 (fv3int_053124)
-b8c33daa4b124f116b0ba1b1386968c4c15b42e4 src/dynamics/mpas/dycore
+3a6d999ab0dbee9f03ab6b9a13dd3b6d9670eb54 src/hemco (hemco-cesm1_2_1_hemco3_6_3_cesm_rme)
 825e7f20e2dd368b95b1e3cb2562ab571318bb4d src/physics/ali_arms (ALI_ARMS_v1.0.1)
 bf165cd84ef94087d9a5669a5ad47838ab24c0ef src/physics/carma/base (carma4_01)
-50cee042e588fefd3fe58c2a1d638affec8c0389 src/physics/clubb
-34d8eef3d231a87c0f73e565f6b5d548876b294a src/physics/cosp2/src
 84f27d804207e79e344e8deec98b471207f9b1f0 src/physics/pumas (pumas_cam-release_v1.36)
 be3cad3a12d25918f5016b509b15057f84aab608 src/physics/pumas-frozen (pumas_cam-release_v1.17_rename)
 df02975ab93165b34a59f0d04b4ae6148fe5127c src/physics/rrtmgp/data (v1.8)
 4d8c5df4c63434aaab854afd1b02f5986d41dfb3 src/physics/rrtmgp/ext (v1.7)

But even the top-level git-fleximod does not reinit the submodules:

cd ../..
./bin/git-fleximod update

This gives the output:

➜  CESM git:(master) ✗ ./bin/git-fleximod update
            ccs_config at tag ccs_config_cesm0.0.109
                 share at tag share1.0.19
                  cime at tag cime6.0.246
                   mct at tag MCT_2.11.0
            mpi-serial at tag MPIserial_2.5.0
s                  cam cam6_3_162-41-ga1261980 a126198071fe4e831cd1465d44e6eb0dd3f9cb1a is out of sync with .gitmodules cam6_3_162
                   ww3 at tag ww3i_0.0.2
                   rtm at tag rtm1_0_79
                pysect at tag 3.2.2
                mosart at tag mosart1_0_49
             mizuroute at tag cesm-coupling.n02_v2.1.2
                   fms at tag fi_240516
            parallelio at tag pio2_6_2
                 cdeps at tag cdeps1.0.37
                 cmeps at tag cmeps0.14.63
                  cice at tag cesm_cice6_5_0_9
                  cism at tag cismwrap_2_2_001
                   clm at tag ctsm5.2.007
                   mom at tag mi_240522
          ccs_config up to date.
               share up to date.
Recursively checking out submodules of cime

e    CIME/non_py/cprnc has no fxtag defined in .gitmodules, module at v1.0.6
                cime up to date.
                 mct up to date.
          mpi-serial up to date.
Recursively checking out submodules of cam

...
              geoschem at tag 14.1.2
                 hemco at tag hemco-cesm1_2_1_hemco3_6_3_cesm_rme
...
Recursively checking out submodules of hemco

e                HEMCO has no fxtag defined in .gitmodules, module at 3.6.3-cesm
                 cam up to date.

The underlying submodules in src/chemistry are never updated.

lizziel commented 4 days ago

@jimmielin, did you also do a final step of running fleximod within CAM? I would think this would then update the submodules. Or are you saying you did this and it doesn't work?

jimmielin commented 3 days ago

Yes, running fleximod within components/cam gave the same result. So somehow fleximod will not recurse and reinitialize git submodules unlike git submodule update --init --recursive. Maybe there is a flag for this behavior that I am not aware.

jedwards4b commented 3 days ago

Okay - the minimal example helps - and you are correct, that isn't the expected behavior. Here is what I think happened: It looks like you updated the cam/.gitmodules file to reflect the new modules that you want for cesm but when you committed that branch you didn't update the corresponding submodules. You can see this with a standalone checkout of your cam branch:

 git clone https://www.github.com/geoschem/CAM -b feature/geoschem_14.4 cam_geoschem_14.4
Cloning into 'cam_geoschem_14.4'...
warning: redirecting to https://github.com/geoschem/CAM.git/
remote: Enumerating objects: 39126, done.
remote: Counting objects: 100% (231/231), done.
remote: Compressing objects: 100% (79/79), done.
remote: Total 39126 (delta 159), reused 207 (delta 152), pack-reused 38895
Receiving objects: 100% (39126/39126), 46.56 MiB | 18.75 MiB/s, done.
Resolving deltas: 100% (28561/28561), done.
Updating files: 100% (2198/2198), done.
derecho7: ~/sandboxes
:) cd cam_geoschem_14.4/
derecho7: ~/sandboxes/cam_geoschem_14.4
:) git status
On branch feature/geoschem_14.4
Your branch is up to date with 'origin/feature/geoschem_14.4'.

nothing to commit, working tree clean
derecho7: ~/sandboxes/cam_geoschem_14.4
:) ./bin/git-fleximod status
e            chem_proc not checked out, aligned at tag chem_proc5_0_06
e                carma not checked out, aligned at tag carma4_01
e                pumas not checked out, aligned at tag pumas_cam-release_v1.36
e         pumas-frozen not checked out, aligned at tag pumas_cam-release_v1.17_rename
e             ali_arms not checked out, aligned at tag ALI_ARMS_v1.0.1
e           atmos_phys not checked out, aligned at tag atmos_phys0_02_006
e                  fv3 not checked out, aligned at tag fv3int_053124
e             geoschem not checked out, out of sync at tag None, expected tag is geoschem_14.4_in_cesm
e              cloud_j not checked out, out of sync at tag None, expected tag is feature/set_levels_with_cloud_from_parent_model
e                 hetp not checked out, aligned at tag geoschem_hetp_1.0
e                hemco not checked out, out of sync at tag None, expected tag is dev/hemco-cesm1_4_0_hemco3_9_0
e           rte-rrtmgp not checked out, aligned at tag v1.7
e          rrtmgp-data not checked out, aligned at tag v1.8
e                 mpas not checked out, aligned at hash b8c33daa
e                cosp2 not checked out, aligned at tag v2.1.4cesm
e                clubb not checked out, aligned at tag clubb_4ncar_20231115_5406350
e                 cism not checked out, aligned at tag cismwrap_2_2_001
e                  rtm not checked out, aligned at tag rtm1_0_79
e               mosart not checked out, aligned at tag mosart1_0_49
e            mizuRoute not checked out, aligned at tag cesm-coupling.n02_v2.1.2
e           ccs_config not checked out, aligned at tag ccs_config_cesm0.0.106
e                 cime not checked out, aligned at tag cime6.0.246
e                cmeps not checked out, aligned at tag cmeps0.14.63
e                cdeps not checked out, aligned at tag cdeps1.0.34
e                share not checked out, aligned at tag share1.0.19
e                  mct not checked out, aligned at tag MCT_2.11.0
e           parallelio not checked out, aligned at tag pio2_6_2
e                 cice not checked out, aligned at tag cesm_cice6_5_0_9
e                  clm not checked out, aligned at tag ctsm5.2.007
e                  fms not checked out, aligned at tag fi_240516
    testfails = 3, local mods = 0, needs updates 30

To try to solve this I created a copy of your branch in my fork https://www.github.com/jedwards4b/CAM
but when I try to checkout geoschem I am getting an error: 'geoschem_14.4_in_cesm' did not match any file(s) known to git' Looking in you .gitmodules file I realize that the fxtag should be feature/geoschem_14.4_in_cesm Okay - so I made that correction, ran git-fleximod update geoschem cloud_j hemco to get the submodules you have changed and then git commit -a to update the submodule pointers in git. I then pushed that branch back to my fork.

You can see in my fork that these submodules now point to the correct hashes: eg: geoschem

Going back to cesm though I see that that still hasn't solved the problem. I'll need to dig into it more and get back to you.

jimmielin commented 3 days ago

Thanks @jedwards4b. Yes, everything works if we git clone https://github.com/geoschem/CAM, and use CAM as the top-level. But when CESM is the top level it doesn't recurse down to update the submodules.

Yes, geoschem_14.4_in_cesm is misspelled in the .gitmodules file in geoschem/CAM, it should be feature/geoschem_14.4_in_cesm. Thanks for noticing - I will let @lizziel know. This is a temporary fork used for development, in any case.