Code reuse: git submodules vs superbuild vs other methods

[jgvictores]

In https://github.com/robotology/QA/issues/156 we commented on the intention of splitting several https://github.com/roboticslab-uc3m projects into mini-repositories and then unifying them through git submodules. An alternative was presented (in addition to another discussion), the superbuild mechanism.

I have personally always been quite reluctant to git submodules. On the other hand, I have usually been satisfied with solutions proposed by the robotology community, and will be as long as YCM is just a thin layer over CMake proves to be YCM is just a thin layer *within* CMake as seems to be the fact.

@PeterBowman @David-Estevez Could you please share your opinion on this issue with us?

[David-Estevez]

In my experience, git submodules are a good idea with a somewhat not so good implementation.

Let's imagine we have two projects: Parent and Child and, as Parent depends on Child, we want to convert it to a git submodule. When using a git submodule we have to specify in the Parentrepository which version (i.e. commit) of Child is the one we are using within Parent. Up to this point, everything seems ok.

Complications arrive when we have several git submodules in our project and they are quite active. Each time we update one of those submodules, such as Child, we have to make a commit in Parent specifying that we want to use that new version. If we have several projects that depend on Child, we have to update all of them by hand, otherwise they will use the old version. This means that if we have a submodule that is updated quite frequently (as our repositories) and if it is used by several project (as our repositories) it is quite a mess to maintain all the projects up to date.

Therefore I would discourage using git submodules if there is an alternative suitable solution. I've never used superbuild, but it seems to wrap the cmake dependency system to download and compile external projects, which seems to be a better idea for rapidly changing projects.

[PeterBowman]

I can see one major advantage in using git submodules: prevention from compatibility issues. Let's say repo A links to repo B, which hosts a library with high contribution traffic. At some point, contributors to B make a breaking change that is shipped to end users (suppose no release notes are supplied, nor read otherwise). This should never happen in common scenarios thanks to proper versioning of software (and well-documented deprecations), but I wouldn't discard it in a quickly changing environment. In this cases, I'd deem pointing to specific commits as a preferable solution. Apart from that, I agree with David's remarks and this overview of the situation.

I had never the chance to use superbuild, but I'm a bit skeptical about its utility to developers, at least until it's proven to behave well on different IDEs and Git clients. However, it might be a great helper for users that just want to download and compile code; maybe a separate repo could be maintained with this aim, compare with https://github.com/robotology/superbuild-example. On the other hand, YCM's build system might fit nicely in our current workflows (I'm not sure yet to which extent, if at all, but maybe it could alleviate these issues and prevent us from doing this).

I came up with a quite simple idea of gathering all non-robot specific code in a single repo that would act as a suite of libraries and programs. That is, reducing a quite high number of repos to a single one. How could this work in practice?

[jgvictores]

I came up with a quite simple idea of gathering all non-robot specific code in a single repo that would act as a suite of libraries and programs. That is, reducing a quite high number of repos to a single one. How could this work in practice?

I've begun #2 to work on this, and focus this issue on the mechanism to integrate the resulting repositories. Focusing on the mechanism to integrate the resulting repositories... any volunteer to try out superbuild?

[PeterBowman]

As a side note, YCM 0.2.0 has been released today.

[RaulFdzbis]

Hi!

It looks, like Ill be the one in charge of try out that thing called "superbuild" :D.

I have been talking with @jgvictores, and we will try it with the https://github.com/roboticslab-uc3m/teo-openrave-models and this one https://github.com/roboticslab-uc3m/xgnitive. Our final goal will be to load the model from teo-openrave-models in the xgnitive repository using superbuild.

I will keep you up with any advances!

[RaulFdzbis]

So, i will do a fast sum up of my first impressions about how superbuild works. First its completely integrated with Cmake, this means that is easy to add and use in already existing projects. Good news.

Without enter in detail, how superbuild works, is in the following way. It creates a new folder (named superbuild) in the root of the repo. Then, in this new folder it clones (using github, svn...) the repos added to the system with superbuild. Finally these sub-repos are then compiled in the "build/superbuild" folder.

[David-Estevez]

@RaulFdzbis It is possible to change that path?

I mean, suppose I already have that repo cloned in the same folder as the parent repo (because I'm working on it)... does superbuild need to download it again in the superbuild folder?

[jgvictores]

@David-Estevez From what I've seen, it does some kind of check to see if the dependency is already installed (and therefore avoid download). Have to check exact behavior.

If we finally go for this mechanism, we have to be sure to add the superbuild it generates in .gitignore, otherwise we'd be in big trouble.

[RaulFdzbis]

@David-Estevez yes Juan is right. First it checks if the repo is already downloaded. However, in the tests i did, this first check gave me some errors when the repo was already downloaded. To fix this, what i had to do was to change the name of the cloned sub-repo in the superbuild structure, in order to avoid superbuild to find the already downloaded repo. So this may need a little work around.

[David-Estevez]

So it first downloads the dependency, and then checks if it is already present in the system?

[RaulFdzbis]

First it checks if it is already in the system, and then if it is not superbuild download it.

[jgvictores]

A small comment just to point out some major use cases where code could and should be reused:

• gtest
• ColorDebug
• travislib

[PeterBowman]

A small comment just to point out some major use cases where code could and should be reused:

gtest
ColorDebug
travislib

The following repos have been imported/migrated to our organization:

• color-debug
• travislib

I think gtest may fall under the use case of YCM soft dependencies.

[jgvictores]

Maybe also...

• installation-guides

[munozyanez]

Some people asked me to share the script I use to automate submodules. I can not figure if superbuild is being implemented, but in any case, I made a gist to share the code and receive comments. Find it here: https://gist.github.com/munozyanez/1f0fd624ed17567e01478ca301e1b199

[jgvictores]

?

[RaulFdzbis]

@jgvictores could you explain better your point?

[jgvictores]

@RaulFdzbis Keep it real!

[jgvictores]

Not sure if related, but maybe someone is interested at taking a look? ( @PeterBowman ;wink;wink; ) https://docs.hunter.sh

[jgvictores]

Updates on this are that YCM is pretty cool, but:

1. Problems when offline.
2. AFAIK, no-one is really using teo-main which is a pure YCM superbuild, not sure why. Bump https://github.com/roboticslab-uc3m/teo-main/issues/2
3. More YCM stuff such as the hash clash thing identified by @PeterBowman (a bit in a hurry to find related YARP issue).

[PeterBowman]

More YCM stuff such as the hash clash thing

a bit in a hurry to find related YARP issue

We can actually handle this regardless of what YARP folks decide to do, see #24.

[PeterBowman]

AFAIK, no-one is really using teo-main which is a pure YCM superbuild

Just noting that the newly refurbished asibot-main and amor-main repos will soon grow the list of YCM-driven projects.

[PeterBowman]

Not sure if related, but maybe someone is interested at taking a look? ( @PeterBowman ;wink;wink; ) https://docs.hunter.sh

I've recently heard of Conan, a C/C++ package manager: https://conan.io.

[PeterBowman]

Bump. I think that YCM is already well established and is proven to work fine in the usual superbuild scenario (teo-main, soon asibot-main and amor-main will follow). Moreover, it copes well with header-only repos, and well enough with CMake files; in that respect, color-debug is already cloned and configured on-demand in several repos. It is not the preferred way to build from source code in non-superbuild repos; therefore, 3rd-party dependencies like the googletest framework are discarded ATOW (#41).

That being said, is it OK to close this issue?

[jgvictores]

Yess!!