search

trilinos / Trilinos

Primary repository for the Trilinos Project
https://trilinos.org/
Other
1.2k stars 564 forks source link

Decide on how to handle multi-repos with Trilinos both short-term and long-term #452

Closed bartlettroscoe closed 3 years ago

bartlettroscoe commented 8 years ago

Next Action Status:

???

CC: @trilinos/framework

Blocking: #440, #176

Description:

When Trilinos was trimmed down and moved to GitHub, several extra repos were split out into their own repos like MOOCHO, Sundance, CTrilinos, ForTrilinos, Mesquite, etc. There was a plan for managing these extra repos using the support for extra repos added for TriBITS to support CASL VERA development in the Goolge Doc Proposal for trimming down Trilinos repo which was started in late 2014 or so. Given that Trilinos was split up, I (@bartlettroscoe) assumed that the plan listed in that document was a reasonable approach to get these extra repos for inserted packages back into automated testing and for distribution in Trilinos releases (because CASL VERA has been doing this for years). Based on this assumption, I created the issue #176 to get this working for Trilinos, CCing the other Trilinos Framework team members. I think did some work on this approach and got many of these inserted packages back under automated testing and added full support for the usage of the TriBITS clone_extra_repos.py and gitdist scripts. However, my assumption that there was agreement to use the the TriBITS approached used by CASL VERA as an initial way to address multiple repositories in Trilinos was not founded. It seems that others did not agree with that decision.

Therefore, this Trilinos GitHub ticket is to drive and document the discussion of this topic and the decisions on how Triilnos should deal with multiple repos, both short-term and long-term.

bartlettroscoe commented 8 years ago

@trilinos/framework,

To start this off, I would like to propose that since the support for the TriBITS approach is already implemented as described in #176, that it would be reasonable to add a little Trilinos-specific documentation on this approach and then let people evaluate it and provide feedback on the process and tools. Then other approaches like Google Repo (being used by SIERRA) and git submodules can also be evaluated as a possible better long-term official solution. It may even be possible to let developers use either Google Repo or gitdist depending on their preferences (but I would need to learn more about how Google Repo works first). And even if git submodules were selected, a tool like gitdist might also be used by some people to augment that workflow (e.g. using gitdist dist-repo-status, etc.).

Does this sound like a reasonable approach? If not, let's have a meeting to discuss this specific proposal to evaluate the TriBITS approach as an evaluation (not a as the decided-upon long-term solution).

jwillenbring commented 8 years ago

My opinion is that it is reasonable to put together some Trilinos-specific documentation for gitdist, provided that isn't more work than it sounds. I would like to offer an alternative to the follow-on, in that we could contact Paul W and see what his impression is of Google Repo. If it is favorable, we could see how much work it would be to put together just a few bits of documentation on that too, so we could have a reasonable comparison. I think we would want to do at least a preliminary framework review of the tools, then potentially extend the evaluation to interested developers (I like that idea). If Google Repo doesn't seem like a reasonable option, we would need to reevaluate next steps. I personally find it unlikely git submodules will be the best solution, though we could talk about this if warranted. Given what I know now, I would be satisfied to not include this in the (serious) discussion. I also want to consider requirements here, but regardless of what specific models Trilinos is moving too, it seems the requirements will be similar. The potential big shift I see coming is that instead of multiple repos being the exception, it may become required.

bartlettroscoe commented 8 years ago

Responding to @jwillenbring for above comment:

My opinion is that it is reasonable to put together some Trilinos-specific documentation for gitdist, provided that isn't more work than it sounds.

Should not take more than a hour to write up some documentation. I would also like feedback on the tools themselves. For example, @bmpersc mentioned that he did think the current gitdist documentation (e.g. gitdist --help) was very effective. I would like to address that and incorporate that feedback into the gitdist tool, for example. Since CASL VERA is using gitdist, I can fund that specific work with CASL, not IC funds.

I would like to offer an alternative to the follow-on, in that we could contact Paul W and see what his impression is of Google Repo. If it is favorable, we could see how much work it would be to put together just a few bits of documentation on that too, so we could have a reasonable comparison.

I 100% agree. If nothing else, perhaps we could just get access to the SIERRA documentation on the usage of the Google Repo tool? The main documentation for Google Repo that I found online here does not really describe the full workflow for setting it up and using it. From that documentation it sounds like they are forcing a topic-branch workflow. From looking at this documentation, the setup to use it seems a good bit more involved than for gitdist and it would impact all Trilinos developers, not just those who want to work with extra repos. And that documentation does not even describe a full development workflow, just how to clone the repos. I have no idea how to actually use this tool from any documentation that I can find (but I can perhaps guess).

I think we would want to do at least a preliminary framework review of the tools, then potentially extend the evaluation to interested developers (I like that idea). If Google Repo doesn't seem like a reasonable option, we would need to reevaluate next steps.

Agreed.

I personally find it unlikely git submodules will be the best solution, though we could talk about this if warranted. Given what I know now, I would be satisfied to not include this in the (serious) discussion.

Mark Hoemmen mentioned that someone was trying to sell him on Trilinos usage git submodules and he asked me about it (and I gave him an ear-full). Because if that, I think such an evaluation has to describe the technical issues and the usability problems of why we would not evaluate git submodules further. People are always going to as after the fact "why not use git submodules" otherwise and we need to have a response (which will be documented at the very least in this Issue ticket).

I also want to consider requirements here, but regardless of what specific models Trilinos is moving too, it seems the requirements will be similar. The potential big shift I see coming is that instead of multiple repos being the exception, it may become required.

That is a good point. But it will be a big shock to some people once we make a switch like that.

BTW, I want to use what we learn as part of this evaluation to write the document "Compare and contrast different technical approaches for multi-repository development and integration" for the IDEAS HOW-49 issue. Both you and Brent (and Paul W. if he is interested) can be co-authors if interested.

bartlettroscoe commented 8 years ago

So Google Repo ties you to the usage of Gerrit?

https://news.ycombinator.com/item?id=3904932

jwillenbring commented 8 years ago

I am potentially interested in co-authoring the IDEAS HOW-49 document.

I followed up with Paul W. and Jim F. on 6/17 concerning their use of Google Repo. They provided useful links and comments to assist in evaluation.

bartlettroscoe commented 8 years ago

Here is some better documentation for the Google Repo tool that explains the manifest XML file better:

http://blog.udinic.com/2014/05/24/aosp-part-1-get-the-code-using-the-manifest-and-repo/

The schema of the manifest XML file is given here:

https://gerrit.googlesource.com/git-repo/+/master/docs/manifest-format.txt

From what I have seen of the Google Repo tool today, the main attraction is the fullness and flexibility of the manifest XML file and then then repo sync command to clone all of the repos. Google Repo's manifest file (and the fact that you can specify local manifest files that amend the version controlled one) is much more powerful and flexible than what is supported by the git submodules command, for example. It is also much more flexible than what is currently supported by TriBITS.

But the actual local commands other than repo sync of the repo tool don't look all that great. Looking at the SNL project's page on Google Repo, I would say that the usage of gitdist would look a bit cleaner and simpler and actually more powerful. We will have to do a more detailed evaluation and show a full realistic comparison on Trilinos to show the differences. So if I was not using TriBITS, I would use Google Repo tool to manage the list of repos and clone the repos but use the gitdist tool to run commands after the repos were cloned.

But TriBITS needs to maintain a list of TriBITS repos anyway so there is a lot of duplication between the TriBITS ExtraRepositoriesList.cmake file, the clone_extra_repos.py tool, and the <Project>RepoVersion.txt file and the Google Repo manifest file and the Google Repo repo sync command. The question is if Trlinos needs the extra flexibility as provided in the Google Repo manifest file and for what extra flexibility that is needed by Trilinos if that is easy or hard to add to TriBITS tools.

The Google Repo manifest XML file serves in the roles:

But note that TriBITS allows you to replace the ExtraRepositoriesList.cmake file through the <Project>_EXTRAREPOS_FILE variable so this together with a <Project>RepoVersion.txt file gives almost identical capability of the Goolge Repo manifest file.

But setting up and using the Google Repo tool with the manifest files looks a good bit more complicated than using git submodules and TriBITS for that matter. For example, you have to maintain a separate git repo just for the manifest files.

With a little more research, I think I could list out the major pros and cons of TriBITS multi-repos support and Google Reppo (just with respect to git version control and nothing else).

bartlettroscoe commented 8 years ago

I downloaded the most current version of the Google Repo tool with:

$ curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo

The updated script seems to be much more sophisticated than the version that is documented at:

For starters, there is a bunch more commands:

$ repo help --all
usage: repo COMMAND [ARGS]
The complete list of recognized repo commands are:
  abandon        Permanently abandon a development branch
  branch         View current topic branches
  branches       View current topic branches
  checkout       Checkout a branch for development
  cherry-pick    Cherry-pick a change.
  diff           Show changes between commit and working tree
  diffmanifests  Manifest diff utility
  download       Download and checkout a change
  forall         Run a shell command in each project
  gitc-delete    Delete a GITC Client.
  gitc-init      Initialize a GITC Client.
  grep           Print lines matching a pattern
  help           Display detailed help on a command
  info           Get info on the manifest branch, current branch or unmerged branches
  init           Initialize repo in the current directory
  list           List projects and their associated directories
  manifest       Manifest inspection utility
  overview       Display overview of unmerged project branches
  prune          Prune (delete) already merged topics
  rebase         Rebase local branches on upstream branch
  selfupdate     Update repo to the latest version
  smartsync      Update working tree to the latest known good revision
  stage          Stage file(s) for commit
  start          Start a new branch for development
  status         Show the working tree status
  sync           Update working tree to the latest revision
  upload         Upload changes for code review
  version        Display the version of repo

Note that the script ‘repo’ that you download is just a shell of a script. The guts of the script are only downloaded using the ‘repo init’ command. That command creates the directory ./.repo. Therefore, the size of the ‘repo’ script is deceptive.

Comparing the current ‘gitdist’ tool (which really is just one python script):

$ cd TriBITS/
$ ./common_tools/cloc/cloc.pl tribits/python_utils/gitdist.py
       1 text file.
       1 unique file.                              
       0 files ignored.

http://cloc.sourceforge.net v 1.55  T=0.5 s (2.0 files/s, 2910.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1            300            521            634
-------------------------------------------------------------------------------

to the size of the of the Google repo tool:

$ $TRIBITS_BASE/TriBITS/common_tools/cloc/cloc.pl ~/bin/repo .repo/
     218 text files.
     216 unique files.                              
     160 files ignored.

http://cloc.sourceforge.net v 1.55  T=1.0 s (58.0 files/s, 12347.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          46           1708           2357           7555
Bourne Shell                    12             87            230            410
-------------------------------------------------------------------------------
SUM:                            58           1795           2587           7965
-------------------------------------------------------------------------------

you see that the Google Repo tool is about 12x larger than the gitdist script.

bartlettroscoe commented 8 years ago

At the Trilinos Leaders Meeting today, Siva (@srajama1) pointed to a recent article:

on how Google maintains their entire online application suite (i.e. Google search, GMail, Google Drive, etc.) in a single monolithic version control repository on a single "HEAD" branch using their home-built Piper VC tool. Some notes from that article are given below.

Looks like Google is putting out other articles on Piper as well like:

There are a few takeaway points that I have derived from this article:

Perhaps other readers came away with other takeaway points?

I will argue that nothing in this article argues strongly against breaking up a repo into multiple repos. What is does argues for is using continuous integration, single branch development, and code reviews, all of which can all be done with git repos (which CASL VERA proves). We can do that with a moderate set of separate git repos using the approach of treating a set of repos as one big repo (i.e. the approach used for Android with the Google Repo tool and with TriBITS and the gitdist tool used for CASL VERA). I believe that the TriBITS approach and gitdist tool can effectively scale to 100s of git repos (i.e. using gitdist-mod, gitdist-mod-status). I have an idea for how to address atomic pushes and pulls that will resolve most of the current cases where this will break down (see the --use-semi-atomic-multi-pushes option).

Detailed Notes:

Some notes from that article:

Why a single repository and branch makes sense for the Google online application suite:

  1. Google has the resources to create new custom tools that make this possible in a way that most other organizations can't.
  2. The online Google products are all deployed on the same Google Cloud of machines as a complete and compatible suite. There are no other "Internets" that need to be supported.
  3. Google also developed their customized cloud-based Blaze build system for building all of this. That tool is not given out by Google (but they do give out a non-cloud serial version called Basel).

Note that all Google products are not in this single monolithic Piper repo. For example, Google Android is managed as a bunch of smaller git repos using the tool Google Repo. Why are these Google projects not using or moving to a big monolithic Piper repo?

Why a single repository does not make sense for Trilinos:

  1. Trilinos brings together different projects, different funding agencies, and different customers. Hard to do that with a single monolithic repo. At the very least, you need a distributed version-control tool like git. Piper or any monolithic centralized VC system would not be a great choice for Trilinos based on that.
  2. The same Trilinos package can have multiple masters. For example, DataTransferKit has major funding and usage from CASL and NEAMS. Should it be forced into the main Trilinos git repo?

In my mind, what this article really is arguing for is the value of continuous integration and trunk-based development (instead of creating lots of feature branches like some projects are doing with git).

bartlettroscoe commented 8 years ago

The concept of treating a set of git repos "as one repo" seems to be a pretty common idea. See:

Here is a nice overview of some of the tools that exist for managing sets of git respos:

We could evaluate the pros and cons of all of these in great detail but I like the simple idea of just forwarding using basic git commands for most workflows.

bartlettroscoe commented 8 years ago

Looked into the mr tool some. See this comment. I am sure it works but it is not ideal for git developers (e.g. you have name your remote 'origin').

bartlettroscoe commented 8 years ago

Note that an updated version of the git-annex extension to git might be an attractive way to handle large (binary) files with Trilinos repos (see this comment). I think gitdist could be extended to automate the sync process with git-annex repos to make this easier to use. This looks very promising (and might actually be better than Git LFS overall).

bartlettroscoe commented 8 years ago

Note that an updated version of the git-annex extension to git might be an attractive way to handle large (binary) files with Trilinos repos

Git-LFS is easier to use that git-annex and is ready to use right now with GitLab at SNL and GitHub for that matter but it has some downsides. (see TriBITSPub/TriBITS#139 and TRIL-62) But the files that Trilinos has to manage are not all that large so just storing them in a separate regular git repo is likely the best option.

michaelasprague commented 8 years ago

Hello, This is a useful thread -- we're going through a similar discussion with another project. Sounds like you're still working through it. Have you considered git subtree as an alternative to submodule?

bartlettroscoe commented 8 years ago

@michaelasprague, in this case, we really want to keep these directories outside of the main repo for one of the following reasons:

1) Avoid the extra size of the some of these directories 2) Remove none-critical/deprecated packages 3) Allow for more flexible access control for some of these pieces (e.g. some have not been through Sandia R&A to allow external release through GitHub).

In general we use git subtree for some tasks (e.g. it is a nice way to pull a directory out of a git repo to create its own repo) but it is very tricky to manage multi-repo merges back and forth so you have to be super careful. Managing the repos independently massively simplifies moving changes back and forth in different integration scenarios.

github-actions[bot] commented 3 years ago

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. If you would like to keep this issue open please add a comment and remove the MARKED_FOR_CLOSURE label. If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.

github-actions[bot] commented 3 years ago

This issue was closed due to inactivity for 395 days.