Recipes for conda-forge

[pstjohn]

conda-forge would be a great way to distribute binaries for coin-or's optimization tools, since it allows users to 1) install packages quickly from the command line and 2) specify dependencies & versions in a distributable file.

Current recipes

• I currently have a pull request open for IPOPT at conda-forge/staged-recipes#1607. It compiles and tests on mac and linux, but doesn't currently have a windows build.
• There's already a conda recipe for Cbc, at conda-forge/coincbc-feedstock

  Path forward

We'll need to

• decide on standardized naming system for all the coin dependencies (ipopt vs coinipopt?),
• deal with the existing cbc feedstock - it builds with openblas as its dependency, while I would think we'd want to take advantage of the MKL recipes.
• develop a standardized way of creating the recipes so they could be quickly made for each of the different solvers.

  Potential issues

In writing the test script for conda-forge/staged-recipes#1607, I previously ran into some issues linking against the conda-installed library on a mac. Conda doesn't like users setting DYLD_LIBRARY_PATH(conda/conda#1169), since part of the conda-build process is to make sure the compiled libraries are portable (see conda/conda-build#312 for more discussions on that). In my test script, I manually set the rpath for the ipopt library in the executable:

# Test linking against the ipopt library
cd Cpp_example

g++ -I$PREFIX/include/coin  -c -o cpp_example.o cpp_example.cpp
g++ -I$PREFIX/include/coin  -c -o MyNLP.o MyNLP.cpp

# Deal with different linker flags
if [ $(uname -s) == 'Darwin' ]; then
  g++ -L$PREFIX/lib -Wl,-rpath,$PREFIX/lib -lipopt \
    -I$PREFIX/include/coin -o cpp_example cpp_example.o MyNLP.o
else
  g++ -L$PREFIX/lib -lipopt -I$PREFIX/include/coin -o cpp_example cpp_example.o MyNLP.o
fi

./cpp_example | grep -q "Optimal Solution"

If this is a requirement going forward, that might have to get added in the documentation for the conda packages.

[tkralphs]

Thanks for getting this rolling @pstjohn. You indeed raise a lot of good points that need to be thought about going forward. I'm pulling @whart222 into the conversation and we'll see if we can move this along.

[pstjohn]

Any updates or additional thoughts on this? Aside from the naming scheme, we could implement the other changes in the feedstock after merging conda-forge/staged-recipes#1607

[tkralphs]

Thanks for the ping @pstjohn. @whart222 is unavailable for a few weeks, so maybe we can get started without him. As we discussed previously, it would be good to use a unified approach to conda recipes for all projects. As we are doing with the homebrew recipes, building up according to a topological ordering of the dependency graph would be good. Since Ipopt is one of the projects without any dependencies, it's probably a good one to start with. I don't know that much at all about conda or the way the recipes work, so I either need to do a little research or just ask some questions here and you can answer if you know and/or have the time.

1. The most important question is whether conda has a dependency mechanism. Are we building each project entirely from scratch, including all depenencies? Or can we pull in dependencies by conda installing them? If the former, then using the build script makes sense. Otherwise, it may not.
2. A second question is whether there exist or it makes sense to create conda recipes for the third party packages rather than fetching them and building them as part of the build process of a particular projects. I think this should be avoided if possible, since for the purposes of distribution, this creates a project that uses multiple licenses.
3. Who is considered to be the entity distributing the software in the case of a conda installation? Does anyone check licenses to make sure redistribution in binary form is actually allowed?

For the homebrew recipes, we are currently not using the build script, since each project is built stand-alone and others are pulled in by brew install. Also, all third party dependencies are packaged separately.

[pstjohn]

The recipe documentation isn't that great, I agree. One of the better resources I found was here: http://conda.pydata.org/docs/building/recipe.html. But I'm definitely happy to help where I can.

1. It does have a dependency mechanism, thats actually how I got interested in building a recipe for ipopt 😄 (as its a dependency for another package). In the meta.yml file, you include a list of conda packages you need for both the build step and the run step.
2. I'd check here for the third party dependencies. For IPOPT, it seemed to want specific versions of the various codes it pulls in, so I left the ASL and Mumps routines intact. I'm sure we could set up conda recipes for each though.
3. I'm not sure on the licensing issues. You are supposed to upload a LICENSE file (and specify the license in the meta.yml), I'm assuming the conda-forge maintainers check that prior to merging. Probably makes sense to check that carefully though.

For the build script, does it download the source code for you? Conda wants you to specify a file to download to start the recipe, along with an sha checksum. In my case I'm pulling from http://www.coin-or.org/download/source/. You are certainly able to upload your own files with the recipe though (I for instance have to patch the third party scripts not to use wget).

[tkralphs]

@pstjohn, thanks for the explanations. With the dependency mechanism, the build script probably isn't needed (although it may be convenient to use it in the end, we will see). We can probably just follow the same basic setup as the COIN-OR homebrew recipes for the most part.

The build script doesn't check out the main project, just the dependencies, so pulling the source as a ZIP file is fine for releases. For pulling source, I think you want to be pulling from here:

http://www.coin-or.org/download/pkgsource/

rather than here. The difference between the two is that the pkgsource includes only the Ipopt subdirectory and not the root directory. This source distribution is meant specifically for packaging a project without depencenies, such as would be done in OS X homebrew or in Linux distros.

Currently, all COIN projects in the Optimization Suite are organized according to a scheme that depends strongly on the SVN externals mechanism for pulling in dependencies. Each project has a root directory that only contains a configure script whose job it is to figure out what projects are present and recursively build them. The actual source and a separate configure script specific to the project itself is contained in a subdirectory that is what gets pulled in when the project is a dependency of some other project. Does that make sense?

Because git doesn't have the same notion of externals or the ability to (easily) check out subdirectories, as well as a host of other reasons, we are in the process of eliminating the common root directories for projects and eliminating the use of externals. The build script is actually meant to replace the externals mechanism and the functionality of the root configure script.

With that said, if you download the source from here, then the third party projects will no longer be automatically detected and built by running configure, since this was being done by the configure script in the root directory. If we still need to build some third party projects from source using the COIN-OR build harness, then we could use the script for that. However, I hope we can avoid that.

As far as the problem of matching versions when using the existing conda recipes for third party dependencies, this may not be a big problem. In a lot of cases, the specific versions in COIN-OR are only pegged because no one has bothered to update them, not because of incompatibility :). In the homebrew recipe for Ipopt here:

https://github.com/Homebrew/homebrew-science/blob/master/ipopt.rb

it looks as though the most recent version of Mumps, for example, is being pulled in. This is currently the 5.0 version and seems to be working fine. Maybe you can take a look at what is being done in homebrew and try to follow that to be more or less the same? Let me know if you have questions.

[pstjohn]

Gotcha, that makes things a lot more clear. It looks like there is an in-progress recipe for MUMPS 5.0.1 (conda-forge/staged-recipes#1445), so we might be able to leverage that for a package-only IPOPT install. It doesn't look like there's an ASL package though. This isn't a feature I use often, is that still a recommended dependency? Or are you supposed to build the interface after the fact using something like https://github.com/ampl/mp?

[tkralphs]

ASL is definitely needed, since otherwise, one does not get the stand-alone Ipopt executable, which limits the usefulness of the package. It might actually be good to have an ASL recipe (there is one in Homebrew), but otherwise, we could just download and build ASL as part of the recipe.

[pstjohn]

Sounds great, work in progress at conda-forge/staged-recipes#1778

[pstjohn]

@tkralphs, would you happen to know what license ampl/mp is under? It looks like HPND but thought you might be more familiar.

[pstjohn]

So the first recipe, IPOPT, was merged successfully. Only for Mac/Linux builds at this point, but a windows build could be added into the same feedstock via a PR. (conda-forge/ipopt-feedstock#1).

[tkralphs]

@pstjohn I'm not sure of the provenance of the AMPL license. I can certainly ask. There are three licenses in the included LICENSES file, two of which are almost identical and the third of which is similar. It's unfortunate that it's a bit unclear, but all three seem to be some variant of HPND or MIT. Your guess is as good as mine. Should I inquire? Maybe just including @ampl will do the trick :).

By the way, I guess you saw that I did modify the trunk/master version of the COIN-OR Mumps project to apply the patch we discussed and also changed Ipopt trunk/master so that it includes the mumps_mpi.h header file. Similarly for Clp. Now that I'm thinking about it, for compatibility, what I maybe should do is just check what version of COIN-OR Mumps is being used and adjust the header file name appropriately. I'll push that out to release when I get some time. I guess this doesn't really affect the Conda recipe, though.

I still have on my TODO list to start working on recipes for other COIN-OR projects. Contact me off-line if you're interested in continuing to help with this effort.

[pstjohn]

That license inquiry was from a few months ago, we've got it listed as HPND on conda-forge/ampl-mp-feedstock. I can certainly correct it if they get back to us with something different though.

On modifying the trunk, that's great news: when we bump the version for conda, we could remove https://github.com/conda-forge/ipopt-feedstock/blob/master/recipe/mumps_mpi.patch

On buidling recipes for other projects, I'm certainly happy to help. Its mainly just the time-consuming process of getting the PR to build on the CI infrastructure and get merged by the conda-forge admin team. The build script really isn't that difficult. Speaking of which, do you have a good starting point for 'bld.bat' windows build script for ipopt?

[tkralphs]

A lot has actually happened on this front and there are still plans for more, just the available bandwidth is pretty limited. I think most of the issues you raised initially have actually been resolved. From the fact that you closed as "not planned," though, it seems it doesn't look that way to you. Are there specific things still to be addressed other than building out the set of recipes to include more solvers (mainly Bommin and Couenne are remaining)? If so, maybe a better place to open a related issue would be https://github.com/coin-or-tools/homebrew-coinor.

[pstjohn]

Whoops, I was just going through my backlog and closing out some stale issues I had opened. No offense intended, I figured closing as not planned was a better default. I'm not waiting on anything, in fact I think I'm an (inactive) maintainer of the ipopt conda-forge recipe. I'll close as completed