Submodules to help with inexperienced users

HelixSpiral commented 2 years ago

I saw issue #31 regarding submodules and I was wondering if you were open to using them in combination with your current setup to make things easier for new users.

For example the current setup has users running a shell script to checkout multiple repos into certain directories but unless a user installs the mu tool or manually goes into each repo and checks out a specific commit or tag there's no easy way to sync all the repos up to the same version.

The install could even fail if repos were slightly out of sync when a user ran the shell script (think: a change pushed to main, but something in common or extras breaking because the update to those repos hadn't been merged yet.)

Git submodules would help "tie" things together for an inexperienced enduser, and your current workflow of using mu would be largely unchanged. You could still use mu to update all the sub-repos as normal, however when creating a release tag in the main seiscomp repo you should update all the submodules first, commit that update to the seiscomp repo, and then create the tag, this is to ensure that the release version has the information for where each repo should be at the time of release.

For an inexperienced user this is easier than having to install another tool to checkout and manage a repo, the workflow for an initial install would be something like this: git clone https://github.com/SeisComP/seiscomp cd seiscomp git submodule update --init

gempa-jabe commented 2 years ago

I agree. At the same time I don't see the need to support inexperienced users for compilation. Why not take the compiled packages then?

If you would like to propose a workflow that we can follow to use submodules then we can check it and integrate it when tagging versions. Our daily routine will still incorporate mu as we don't want to switch to submodules. As I understand submodules we would have to commit versions also for the nightly build and that would then duplicate the effort in managing the repositories. With mu we just pull the latest changes of all repositories and build them. No additional commits required. But maybe I am wrong and it is more easy than I think.

HelixSpiral commented 2 years ago

I agree. At the same time I don't see the need to support inexperienced users for compilation. Why not take the compiled packages then?

I was thinking folks that are inexperienced with repo management, but experienced with C++. Not absolutely necessary but it would help make the community more approachable for those people.

If you would like to propose a workflow that we can follow to use submodules then we can check it and integrate it when tagging versions. Our daily routine will still incorporate mu as we don't want to switch to submodules.

I've never used mu, but I think this workflow would work:

cd seiscomp/src/base // Go to base directory
rm -rf main && rm -rf common && rm -rf seedlink && rm -rf extras // Nuke current folders
rm -rf contrib-gns && rm -rf contrib-ipgp && rm -rf contrib-sed
git submodule add https://github.com/SeisComP/main // Add all the submodules
git submodule add https://github.com/SeisComP/common
git submodule add https://github.com/SeisComP/seedlink
git submodule add https://github.com/SeisComP/extras
git submodule add https://github.com/SeisComP/contrib-gns
git submodule add https://github.com/SeisComP/contrib-ipgp
git submodule add https://github.com/SeisComP/contrib-sed
git add main common seedlink extras contrib-gns contrib-ipgp contrib-sed
git commit -a -m "Initial commit with submodules"

Now that all the repos are readded as submodules you should be able to:

Go back to the siescomp repo
Re-register the submodules into mu
mu checkout -b test/feature // Checkout a new branch in the submodules
Make some test changes in any of the submodules
mu commit -a -m "Test commit"
mu checkout master
mu merge test/feature
git status // In the siescomp repo, should show the submodule commit hashes as being updated.
git commit -a -m "Test commit"
git tag v0.0.1
mu push // Push all submodules to github
git push // Push siescomp repo to github

The first set of steps is to delete your current folders and replace them as submodules. If you have any pushed code or things in git stash those should be backed up prior to doing this, or a new repo should be cloned to do this in.

The second set of steps is an example workflow of how mu and submodules could work together. (Note: I don't use mu so my commands for that might be slightly off)

As I understand submodules we would have to commit versions also for the nightly build and that would then duplicate the effort in managing the repositories. With mu we just pull the latest changes of all repositories and build them.

The submodule approach for this would be git clone --recurse-submodules https://github.com/SeisComP/seiscomp, or if you've already got the repo cloned git submodule update --remote

I don't think this would duplicate effort for managing, but I'm not sure what your exact workflow with mu is and I based this off my best guess.

gempa-jabe commented 2 years ago

Thank you for the information. The nice thing of mu is that it is just a replacement for git but calls the command passed on each registered repository. So bascially mu pull in the root directory would pull the latest commit of all repositories. mu push will push all changes made to any repository to their corresponding remotes and so on. Sure you can also just use git on a single repository. mu is like for repo in repos; do git [command] <params>; done.

That being said, is it enough with submodules to run git submodule update --remote to get the latest code of all submodules or do we have to forward the hashes in the main repository for each submodule and commit this state?

Example:

$ cd $HOME/src-tree1
$ cd src/base/common
$ echo "..." > README
$ git add README
$ git commit -a -m "Add README"
$ cd $HOME/src-tree2
$ git submodule update --remote

Will src-tree2/src/base/common contain the README? Sure I could try all that myself but documenting it here will help in further discussions.

HelixSpiral commented 2 years ago

You'll need to do a git push in src/base/common after the commit so that repo pushed to github, but yes that workflow will work. You don't have to push the updated submodule hashes in $HOME/src-tree1 every time a submodule updates.

The only time you really need to commit the hashes of the submodules to the main seiscomp repo is just before you do an actual release, so the tag in that repo has the proper submodule information.

gempa-jabe commented 2 years ago

Thank you for clarification. When I tried submodules years ago I struggled with committing the hashes of the submodules for each repo. That felt very uncomfortable at that time. Maybe it has changed since then and I can give it another try. If I have to do it only if a release is being tagged it would be easy going.

HelixSpiral commented 2 years ago

You're welcome!

I didn't start using submodules until early last year so I'm not sure how they behaved before that, but as it currently stands I think they are great.

They function as their own individual repos so if you don't like using git submodule update --remote you can always cd into them and do git checkout branch && git pull too.

When you start evaluating them I'm happy to help answer any questions you have!

SeisComP / seiscomp

Submodules to help with inexperienced users #33