Configuration fails due to missing dependencies

[vitaut]

It is nice to see Cbc repository finally mirrored on GitHub. However, building it fails at configuration step due to missing dependencies:

$./configure
...
checking for COIN-OR package CoinDepend... not given: No package 'cgl' found
No package 'osi' found
No package 'coinutils' found
configure: error: One or more required packages CoinUtils, Osi, and Cgl are not available.
configure: error: /bin/bash './configure' failed for Cbc

[tkralphs]

Yes, this is essentially because of differences between the way git and subversion work. Cbc depends on a number of other COIN projects maintained in separate subversion repos that must either be already installed or for which the source code must be available in order to build Cbc itself. On SVN, checking out Cbc automatically fetches the sources for the dependencies and places them in the standard location in the Cbc source tree. This is accomplished by the use of the subversion "externals" mechanism, which make this completely transparent. Our whole build system was designed around this and it is unfortunately not very easy to replicate the same mechanism with git :(.

What you see mirrored here on github are the contents of the Cbc repository on SVN without the externals (which are not actually part of the repository). There isn't really a very good way to mirror the dependencies of Cbc from their separate subversion repos into the Cbc git repo and still allow the code of all the dependent projects to be easily updated. There are ways of doing it, but none that seem very sane. We would have to mirror static versions of the dependencies and update them manually (by script) somehow and the result would be identical copies of the code for a project like CoinUtils would be stored in a dozen or more git repos. Overall, I think it would be a mess. I have opted (for now at least) to only mirror each project's SVN repo exactly as it is. This means the current system by which you can checkout a project with a single command and build it along with all dependencies does not work on git at the moment.

I spent a good amount of time trying to figure out how to replicate something like the SVN externals mechanism in git, but haven't settled on a solution yet. There are many options, but because of the design of git, it's not really possible to have an exact counterpart of the externals mechanism, however. Git has two mechanisms that are similar: submodules and subtrees. These can be used to replicate the standard structure in a local working repo, but it is not really a solution for the problem you are raising here. After looking into it a bit, I determined subtrees and submodules were not really a good solution in general.

Another alternative would be just to provide a script that builds the projects one by one individually in stand-alone mode. The build harness we use will allow the projects to be built stand-alone if you build them in the right order (essentially, the configure script at the root level or each project does nothing more than ensure the source is present and then build the project in the proper order. If you install all projects into the same prefix, libraries provided by each one will be found automatically when building the subsequent ones. A script something like this should work to get all dependencies of Cbc and build the entire thing (warning: I have not actually run this script, consider it to be a pseudo-code draft).

mkdir COIN 
cd COIN
for i in CoinUtils Osi Clp Cgl Cbc; do
   git clone git@github.com:coin-or/$i
   cd $i
   ./configure --prefix=.. && make install
   cd -
done

This is not so terrible and at the moment seems like the best option. Of course, the above would build all trunks (master branch in git). To build a particular release version, you would need to check out the latest release of the dependencies as well. This would mean the script would need to be specific to a version of Cbc. Perhaps it could be auto-generated by the release script, which should not be terribly difficult to do either.

If you or anyone reading this have an interest in making this happen, I would be happy to help in implementing some solution or other. It doesn't seem like it would be too difficult. Getting builds working from git checkouts is on my TODO list.

[vitaut]

Thanks for a detailed answer, Ted. Could you elaborate why you think Git submodules can't be used for this? It seems SVN externals should map to them pretty well. See, for example, this answer on SO.

[tkralphs]

@vitaut Part of the reason I thought submodules were not a good solution was because it was my understanding at the time (this was a while ago) that submodules worked in the way the original (and now apparently outdated) answer to the SO question you sent described them. It seems the behavior has since been improved so that they can now function something like an svn external.

Despite the improvements, submodules still do not seem to be exactly like an svn external and it seems to me there are still a few issues with using them to mirror dependencies directly into the github repos of dependent projects the way we would want to. As I understand it, what we would have to do to set this up is to first mirror the subversion repo to a local git repo on the COIN server (this is already being done using subgit), then add in the submodules before mirroring to github. In principle, submodules would have to be added for all dependencies of all versions that anyone might conceivably check out, which It seems would require adding perhaps hundreds of submodules. I suppose this could be automated in some way, but we would need to automate the addition of new submodules anytime a new version of any dependency is released. This could probably be done somehow using a combination of post-commit hooks and modified versions of the scripts we use to create new versions, but it would require quite a lot of work to set it up and would probably mean the git repo would be much larger (not sure if this is a real issue or not). In addition, it looks as though users would still need to manually pull updates for dependencies, which would mean they need to be aware of the details of this setup in order to know how to pull updates. With subversion, all of this happens transparently.

My earlier conclusion about submodules still seems valid---they would work best if individual users set the mechanism up themselves as a way of replicating the current directory structure for one particular version (be it trunk, stable, or release). This would come close to replicating the experience of svn externals (except for the need to update all dependencies manually). Doing this would allow a user to track all dependencies within a single git repo, but setting this up centrally as part of the github mirror doesn't really seem too viable. If we make this issue very clear in the README.md and provide clear step-by-step instructions on how to set up the submodules properly for a single version from the Dependencies file that is included in the subversion repo (and contains the current list of dependencies), do you think this would be a good solution? I guess this is more or less what I had in mind that people would end up doing. Do you have a better suggestion?

The whole concept of an external seems to make the most sense with a centralized VCS. With a distributed VCS, there seem to be these kind of unavoidable issues. If/when we switch to git as our primary, the issues will still be the same. Even worse, when mirroring back to subversion from git, it doesn't look like there is a good way to map back the externals property in response to changing versions.

[tkelman]

If you configure in the subdirectory for each project instead of the top level, and you use the same prefix to install everything, and you build in the right order, you should be able to get good results. But you might end up mixing-and-matching different versions than the combination that was presumably tested together for release versions. Decoupling the libraries individually is what's done with the Fedora and Debian packages, and could also be done in a homebrew-coinor tap without too much difficulty, would need to split up current Cbc formula into CoinUtils, Osi, Clp, Cgl, and Cbc. For development versions you can set up --HEAD in Homebrew, would be easy to make that grab trunk/master of every project, but getting something equivalent to our current SVN stables might be trickier.

[tkralphs]

Tony made me realize that my pseudo-script earlier should have been building in the subdirectories for each project, ala

mkdir COIN 
cd COIN
for i in CoinUtils Osi Clp Cgl Cbc; do
   git clone git@github.com:coin-or/$i
   cd $i/$i
   ./configure --prefix=../.. && make install
   cd -
done

As Tony also pointed out, one option we had in mind was a Homebrew/Linuxbrew set up with which one could recursively download and build a project and all its dependencies with one command. I would say that this will happen RSN. There is a homebrew tap set up already and we just need the recipes:

https://github.com/coin-or-tools/homebrew-coinor

[vitaut]

I suppose this could be automated in some way, but we would need to automate the addition of new submodules anytime a new version of any dependency is released. ... This could probably be done somehow using a combination of post-commit hooks and modified versions of the scripts we use to create new versions, but it would require quite a lot of work to set it up and would probably mean the git repo would be much larger (not sure if this is a real issue or not).

I think it's a matter of updating the release scripts. Since the information about dependencies is already encoded as externals this should be pretty straightforward and not more difficult than writing the build script you mentioned in your first post.

Git submodules have little effect on repo size because they are just links to other repos. They will be retrieved when cloning the repo recursively, but Git is more space-efficient than SVN, so the whole Git repo is often smaller than one checkout of SVN (unless you have a huge history).

In addition, it looks as though users would still need to manually pull updates for dependencies, which would mean they need to be aware of the details of this setup in order to know how to pull updates.

All a user needs is to recursively get a repo git --recursive .... I think this is something most Git users already know how to do as submodules are widely used.

If we make this issue very clear in the README.md and provide clear step-by-step instructions on how to set up the submodules properly for a single version from the Dependencies file that is included in the subversion repo (and contains the current list of dependencies), do you think this would be a good solution?

Not really, because this puts the burden of creating submodules which can be easily automated on the users.

Do you have a better suggestion?

Why not put these instructions in a script form and make them a part of release (or mirroring) step? If I understood correctly you already have some script that invokes subgit, so I guess it's a natural thing to extend this script to iterate over SVN externals mapping them to Git submodules.

[tkelman]

@vitaut the main difficulty with mapping SVN externals directly to submodules for the coin-or repos is that the SVN externals between coin-or projects are not the entire repository of the other project. They are only the subfolder. Git submodules can't do this, you can only have a submodule for an entire other repository, not just a piece of it. Subtrees can do this but are basically just importing the entire code of the other project as a commit dump, and don't maintain an active live connection to a separate repository.

[vitaut]

@tkelman Good point, I forgot that SVN repos have this spaghetti structure. I think there are several ways to address this. One is to use two Git mirrors (or one Git repo with two branches) per SVN repo, one for top-level build stuff and another for real code. Another option is to update the build config so that it accepts additional directory level. Not sure how difficult it is to do with Autotools (would have been trivial with CMake).

[vitaut]

As a proof of concept, I wrote this small Python script which creates main branches with the main content in Cgl, Clp, CoinUtils and Osi repos (Cbc dependencies):

import os
from subprocess import check_call

for repo in ['Cgl', 'Clp', 'CoinUtils', 'Osi']:
  # Clone.
  check_call(['git', 'clone', 'git@github.com:ampl/{}.git'.format(repo)])
  # Checkout new branch 'main'.
  check_call(['git', 'checkout', '-b', 'main'], cwd=repo)
  # Remove top-level entries.
  entries = [i for i in os.listdir(repo) if i not in [repo] and not i.startswith('.git')]
  check_call(['git', 'rm', '-r'] + entries, cwd=repo)
  # Move the main content to the top-level directory.
  main_dir = os.path.join(repo, repo)
  check_call(['git', 'mv'] + os.listdir(main_dir) + ['..'], cwd=main_dir)
  # Commit and push.
  check_call(['git', 'commit', '-m', 'Move main content to the `main` branch'], cwd=repo)
  check_call(['git', 'push', 'origin', 'main'], cwd=repo)

Then adding these repos as submodules of Cbc and pointing to main branches allows building Cbc from Git without any changes to the build procedure (other than replacing svn with git):

$ git clone --recursive git@github.com:ampl/Cbc.git
$ ./configure
$ make

I hope something like this can be incorporated into the mirroring script without too much trouble.

[tkralphs]

@tkelman Thanks for pointing out the issue of subdirectories, which I had also forgotten about.

This has been a very good discussion and we may be zeroing in on a possible solution, but I still don't see exactly how to scale this and get past a few remaining sticky points to make it work smoothly with subgit. @vitaut Thanks for the proof of concept---that is sort of what I envisioned an individual user would do and it does seem to be functional for that use case.

The biggest remaining problem I see is with releases. At the moment, subgit automatically generates the tags for the releases from the releases/ directory of each svn repo. If we want to have the tags for the releases behave in git as expected (with appropriate submodules), I think they would have to be created after the submodules are added to the stable branches, i.e., not by subgit. We would have to tag a commit that already includes the submodules at the time of the commit, e.g., 627a02d. But since all the commits are going to be generated by subgit before we add any submodules, they wouldn't serve that purpose. It seems we would only be able to easily create tags pointing to the commit that added the submodules, which would be the tip of each stable branch. This commit would not actually be the same as the latest release, though, since our script for creating releases adds an extra commit to the stable branch right after committing the release that undoes a few changes, such as setting the version number back. Maybe this isn't too important and maybe most users would be happy just to work with the tip of each stable branch, which would make life easier.

If we wanted releases to include submodules, a possible solution would be to create the releases as branches instead of tags and then to tag the tip of each release branch, which could be done, but seems a bit messy and not very "git-like". If git was our primary, I think this would all be relatively easy, but not using subgit. Another option is to abandon subgit and do things a bit more manually from the start. But that is again something that would have to be undertaken by someone.

For stable branches, I can see this working well, but setting it up will probably still require some non-trivial scripting. We need to parse the Dependencies file of each stable branch we want to convert and add the submodules for the corresponding stable versions of other projects, after creating their associated "main" branches. I suppose that would not be difficult for a scripting guru.

Which versions do you think we need to provide this functionality for? Only for the current trunks and current stables, along with all associated releases? If we want to get the implemented anytime soon, I will definitely need some help with it.

[vitaut]

According to subgit documentation:

The following entities are not translated from Subversion to Git: ... svn:externals properties

However, it might be possible to use subgit and introduce submodules afterwards by rewriting the history with git filter-branch. I just did a small experiment and it looks like the latter supports adding submodules. For example, the command

$ git filter-branch --tree-filter 'git submodule add https://github.com/cppformat/cppformat.git format' HEAD

added a submodule throughout the whole history.

Alternatively, subgit can be replaced with something like git-svn with the same history rewrite step.

Which versions do you think we need to provide this functionality for? Only for the current trunks and current stables, along with all associated releases?

At least trunks, so that people could work on the latest version and submit PRs. Releases would be useful too, but to a less extent.

If we want to get the implemented anytime soon, I will definitely need some help with it.

I am not a scripting guru =), but could probably help if Python is OK as I'm not very good at shell scripting.

[tkelman]

I'm not sure submodules are quite as important to worry about and get perfect for releases. It is useful to have the git tag at least for each individual project so we can see what state the code was in for a given release, but for the purposes of building a release the standard tarballs should work fine. We could even mirror the release tarballs here as separate uploads associated with the tags using the github releases feature. The automatic tarballs github makes for tags don't do the right thing with submodules anyway.

One thing that I quickly found when I was writing a bunch of scripts experimenting with this kind of mirroring (at the time I was using subtrees instead of submodules, but the challenges were closely related) is that every COIN project has a slightly different setup in terms of what it uses for externals. Some projects have trunk refer to other projects' trunks, or to externals' stables, or even specific releases in some cases. Similar with stables, I think every project manager has a different personal preference for how they like to have the externals set up. Oh, and the Dependencies file does not always reflect the same information as the svn:externals property.

[tkralphs]

@tkelman I think you are probably right that submodules for releases are not so important as long as tarballs are available somewhere. It's kind of a a shame, though, that we may not be able to exploit Github's very convenient automatic packaging of releases. If that mechanism doesn't respect submodules, I guess it's a non-issue.

As for the differences in the preferences of project managers with respect to how to manage externals, I agree it is a bit inconsistent, but I don't see this as a big issue if the PM is using SVN anyway and is managing externals there. We can do the submodules on github in a more consistent way if we like or we might decide to make them match SVN, which should not be too much trouble.

So where does this leave us? I can try to do something, but realistically, it may be a while. For the time being, we may be able to fairly easily do trunks, since the naming of the branches is consistent across all projects. Just setting things up manually for stables on the projects that are the most widely used should also not be too difficult. @vitaut If you write something in Python, I think that's fine and we can always translate it to bash pretty easily if we want.

[vitaut]

OK, if there are no better suggestions, I'll check if my idea of rewriting history to introduce submodules works with subgit.

[kochv]

Have you guys considered making a "build" repository for each project? Take Cbc for example. You could create an empty full-Cbc repository that contains only one .gitmodules file, and one configure script or Makefile. The ,gitmodules file would pull in the full repositories of all the projects, while the Makefile would basically do what Ted's script does (call the necessary builds in each subrepo). It would pass some path variables to the individual configure scripts that will take care of the folder filtering. For example a path like this

CLP_PATH = $(CURDIR)/Clp/Clp

The lower leve configure script would simply check if the variable is set. If not, it uses the default.

ifneq ($(CLP_PATH), Clp/Clp)
CLP_PATH = Clp
endif

That way you can build them independently, or out of the full repo. For VC++, you only need a .sln file in the root of the full repo that includes the projects. The drawback of this is, of course, the need for yet another Git repo per project. Any thoughts?

[tkelman]

The configure script at the top level for each project pretty much already is exactly that. I don't think autotools is especially flexible about recursing into subfolders at arbitrary user-specified levels though.

Making 2 separate git repos per project will just get confusing for everyone. I think Victor's idea of filtering out only the content in the subfolder and making separate branches for that content would be the easiest thing to automate here. I think the git history will end up being separate and detached between the filtered sub-branch vs the main full mirror, but at least the commit messages and authorship times should match.

[vitaut]

@kochv I do something like that in https://github.com/ampl/coin. The drawback is that you have an additional repo and, if you put the history of all dependencies in there (which I don't), it will grow pretty large. For these reasons I prefer a modular approach with submodules.

[kochv]

I was suggesting to use submodules. The only difference is that in using another repo, you wouldn't regenerate the history. The problem with filter-branch is that it will recreate all you git hashes, and the two histories will not be compatible anymore.

[tkelman]

I'm not sure that's really a problem since these are mirrors anyway.

[kochv]

So you will keep the Git repos read-only?

[tkelman]

For the foreseeable future, yeah. I don't expect the core Coin-OR projects to be moving to git as primary any time soon.

[tkelman]

Some smaller or less-deeply-integrated projects will probably move sooner. CppAD is already using git as primary and instead mirroring the other direction.

Ipopt could probably add submodules for its ThirdParty harnesses already, since those don't have quite the same subfolder problem as the Cbc stack. Those ThirdParty submodules just aren't being added automatically by the mirroring scripts yet.

[kochv]

For those of you that are interested, I wrote a cmake file that pulls the individual Git repos for CoinUtils, Osi, Clp, Cgl, and Cbc, configures them, and builds the libraries. Rather than forcing the dependencies of svn externals into the Git repos, it gets rid of them by using a common install directory when building. This solution is non-invasive. It doesn't change anything in the current Git repositories or SVN structure. Also, it doesn't fiddle with Git history so that patches could still be pushed through the individual repos into SVN, if required. To use it, make sure you have cmake >= 2.8 installed. Copy the CMakeLists.txt into your workdir. In the same workdir, execute the following:

mkdir build
cd build
cmake ..
make

The files are placed in build/install. I tested this on Linux and it works fine. I am still ironing out some issues on OS X, so this is just a first version. It should also work on Windows, but I haven't tested that yet. So for those who like to use this, please give me your feedback. Thanks. Here is the CMakeLists.txt file:

cmake_minimum_required(VERSION 2.8)

include(CheckCXXCompilerFlag)
include(ExternalProject)

CHECK_CXX_COMPILER_FLAG("-std=c++11" COMPILER_SUPPORTS_CXX11)
CHECK_CXX_COMPILER_FLAG("-std=c++0x" COMPILER_SUPPORTS_CXX0X)
if(COMPILER_SUPPORTS_CXX11)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11")
elseif(COMPILER_SUPPORTS_CXX0X)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x")
else()
        message(STATUS "The compiler ${CMAKE_CXX_COMPILER} has no C++11 support. Please use a different C++ compiler.")
endif()

ExternalProject_Add(
    CoinUtils
    SOURCE_DIR ${CMAKE_SOURCE_DIR}/CoinUtils
    GIT_REPOSITORY https://github.com/coin-or/CoinUtils.git
    CONFIGURE_COMMAND ${CMAKE_SOURCE_DIR}/CoinUtils/configure --prefix=${CMAKE_BINARY_DIR}/install  CXXFLAGS=${CMAKE_CXX_FLAGS}
)

ExternalProject_Add(
    Osi
    SOURCE_DIR ${CMAKE_SOURCE_DIR}/Osi
    GIT_REPOSITORY https://github.com/coin-or/Osi.git
    CONFIGURE_COMMAND ${CMAKE_SOURCE_DIR}/Osi/configure --prefix=${CMAKE_BINARY_DIR}/install  CXXFLAGS=${CMAKE_CXX_FLAGS}
)

add_dependencies(Osi CoinUtils)

ExternalProject_Add(
    Clp
    SOURCE_DIR ${CMAKE_SOURCE_DIR}/Clp
    GIT_REPOSITORY https://github.com/coin-or/Clp.git
    CONFIGURE_COMMAND ${CMAKE_SOURCE_DIR}/Clp/configure --prefix=${CMAKE_BINARY_DIR}/install  CXXFLAGS=${CMAKE_CXX_FLAGS}
)

add_dependencies(Clp CoinUtils)
add_dependencies(Clp Osi)

ExternalProject_Add(
    Cgl
    SOURCE_DIR ${CMAKE_SOURCE_DIR}/Cgl
    GIT_REPOSITORY https://github.com/coin-or/Cgl.git
    CONFIGURE_COMMAND ${CMAKE_SOURCE_DIR}/Cgl/configure --prefix=${CMAKE_BINARY_DIR}/install  CXXFLAGS=${CMAKE_CXX_FLAGS}
)

add_dependencies(Cgl CoinUtils)
add_dependencies(Cgl Osi)
add_dependencies(Cgl Clp)

ExternalProject_Add(
    Cbc
    SOURCE_DIR ${CMAKE_SOURCE_DIR}/Cbc
    GIT_REPOSITORY https://github.com/coin-or/Cbc.git
    CONFIGURE_COMMAND ${CMAKE_SOURCE_DIR}/Cbc/configure --prefix=${CMAKE_BINARY_DIR}/install CXXFLAGS=${CMAKE_CXX_FLAGS}
)

add_dependencies(Cbc CoinUtils)
add_dependencies(Cbc Osi)
add_dependencies(Cbc Clp)
add_dependencies(Cbc Cgl)

[vitaut]

Thanks for sharing the CMake config @kochv. I think it's fine for building the latest version of Cbc, but I'm not sure that it will work with past revisions. I agree with @tkelman that there is no reason to worry about history rewriting as these git repos are only mirrors. Moreover, if it is done incrementally the hashes can remain unchanged once the links via sumbodules are established.

[kochv]

It's not just the history. It's more the fact that hard-wiring external dependencies into the source tree (through svn externals or newly nested git submodules) removes a lot of flexibility and space. Moving such dependencies dynamically into the build process instead would make the switch between different SCM and build systems much easier and would use significantly less space on your development machine. But maybe you are right, and the old versions will not build like that. I haven't tried.

[vitaut]

The dependencies are already hardwired via SVN externals. I am only suggesting to translate them into Git format be it via submodules or some other mechanism.

[kochv]

And I suggest not to make the same mistake in the Git world again and moving the dependency into the build process instead. But I don't have a strong opinion on that, I am fine with whatever you guys decide.

[vitaut]

On a second thought, it seems to me that your CMake solution might work with past revisions too, @kochv. We only need to parse Dependencies file and translate SVN URLs into their Git equivalents. Should be pretty easy to do in CMake.

[tkralphs]

@kochv The CMake solution is indeed a nice one, especially since it automatically downloads the source. I guess the same could be done with autoconf using a wrapper Makeflle that downloads the source in the same way. This is very much like what Homebrew does as well. Something like this could make the build system agnostic to the underlying VCS. In the end, we might just decide to get rid of all the project root directories and just use a common root build. This is something I've thought about doing before.

@kochv Actually, an empty repository that is just for building does already exist. I created it years ago for the purpose of downloading and building the entire suite of optimization tools (what I have been calling the COIN-OR Optimization Suite) in one go. The project lives here at the moment:

https://projects.coin-or.org/CoinBinary/browser/CoinAll

The externals for the various versions of this uber-project are versions of all the COIN solvers that have been tested together as a unit using this harness. I haven't mirrored it to github yet, but I could. The purpose of the project was to be able to do build and test for the whole suite, as well as produce installable packages of binaries for all projects rather than having separate bundles of binaries for each project. Every year or so, I create a new bundle of the latest stables of all projects and make releases of the entire thing using this wrapper configure script. In fact, I'll cut release 1.8.0 in the next couple of days.

The configure script detects what projects are present so it can actually function as that empty repository you were talking about for building any project. With the addition of a way to automatically download dependencies (like your CMake script), it could function sort of as a universal installer. This probably wouldn't be a huge amount of work. Of course, we could also generalize the CMake solution in the same way. For now, maybe I will try to generalize your CMake harness first and add it to the CoinAll repo as a way of building the entire suite from git clones. In the meantime, we can think a bit more about what to do in the long run.

[tkelman]

The problem is unless you're making a monolithic binary installer, you very rarely want to build absolutely every project at the same time. CoinAll doesn't have an easy way of saying "just build this one project and its dependencies for me" - unless you already know exactly what those dependencies are and manually delete everything else, I suppose.

I don't see the cmake version as being all that different from the Python or shell versions posted early on. Different story if the projects themselves were all using cmake instead of autotools for their build systems, but that's even further off.

[tkralphs]

@tkelman Yes, what I had in mind was to change the CoinAll build harness so that you could say you wanted to just build a subset without having to know the dependency structure. This could be done with some configure options like configure --build-cbc or something like that. This would download Cbc source from the URL in the CoinAll Dependencies file and then parse the Cbc Dependencies to find all dependent projects in order to download them as well. Finally we would just do the configure, build, and install as usual. I guess this would not terribly difficult to do, but this is kind of doing the same job as Homebrew does, so I'm not sure it's needed or will be useful to that many people.

[mlubin]

I like the solution used by Rehearse (https://github.com/coin-or/Rehearse/blob/master/download_dependencies.sh).

[tkralphs]

For those who are still subscribed to this issue, I finally implemented a script for getting the dependencies of any of the COIN-OR projects currently mirrored to github. A preliminary version is here:

https://gist.github.com/tkralphs/13d4529b4d4e8889f3a6

The script parses the Dependencies file in the main project's repo, after which the dependencies are automatically checked out from github (by git clone) or coin-or.org ( by svn co). Using the SVN option gets you exactly the same result as if you checked the project out with externals using svn.

With git, things are simpler than I thought. Checking out a git repo inside another git repo actually works fine so there is really no need for anything fancy like subtrees or submodules. There doesn't seem to be a way around the fact that project subdirectories have to be two-levels deep, though, unless we want to move directories around manually after the fact. Moving directories around would make the local directory structure different than the remote, and that could make trouble with creating pull requests.

I added an option for so-called "sparse checkouts", which is a way of checking only a subdirectory from a git repo. There's actually not much advantage to doing this, since the project subdirectories are still two levels deep, but it does suppress checking out of all the files in the root directories of the dependencies, which we don't really want anyway.

The script also has an option to automatically build after checkout. If you want to get the source for dependencies and build all in one go, you can do

./get.dependencies.sh fetch build

Note that build also installs right now and there is no test command yet. If you want build and test before deploying,DESTDIRshould work, ala

./get.dependencies.sh fetch --git build --prefix=/usr/local DESTDIR=/scratch/build

This has been tested, but probably still needs some more rigorous use to shake out any bugs.

With all of this, we have a sort of generic solution for using Travis-CI with any COIN-OR project on github, so I whipped up a generic .travis.yml, which I've added to to Dip, SYMPHONY, Cbc, and Clp. Voila, we have automatic build and test on Linux and three flavors of OS X. See status here:

https://travis-ci.org/coin-or

There are a lot of tweaks and enhancements possible here, depending on individual preferences about where things get checked out and built. Feel free to comment at the above gist to make suggestions.

[tkralphs]

Just wanted to follow up and say that this script is now officially released as part of the BuildTools here:

https://github.com/coin-or-tools/BuildTools/blob/stable/0.8/get.dependencies.sh

[tkralphs]

Since this has been in production on Travis and AppVeyor for a while now and seems to be working fine, I'm going to close this issue!

[tkralphs]

There has been further development on this in trunk and I would now recommend using

https://github.com/coin-or-tools/BuildTools/blob/master/get.dependencies.sh