Try out conda-forge

[astrojuanlu]

See https://conda-forge.github.io/#add_recipe.

[whart222]

What does conda forge support that the current test infrastructure does not? Is that just a way of bundling these recipes into a larger ensemble? If so, how do we control the processing of these packages?

[astrojuanlu]

Well, we basically had the same idea (combining several continuous integration platforms, the creation of conda packages and the upload process) but they put way more time and effort and their result is much cleaner. With this setup we're in charge of the whole infrastructure, while if we take advantage of the conda-forge process we just have to maintain the recipe, which is nicer and less burden for us.

[astrojuanlu]

Also, Continuum Analytics is endorsing the conda-forge as the "community" side of their conda packages so I think it's a good idea to jump on the bandwagon :smile:

[whart222]

This sounds like a very nice option. I definitely think it's worth exploring. My understanding is that each of the recipes would be migrated into conda-forge individually. Right? So we could try it out with a few individual recipes.

[astrojuanlu]

Exactly! We can start with the core dependencies first and start growing from there. Let's get #11 merged first and then we can migrate the recipes.

[whart222]

I took a stab at setting up PyUtilib in conda-forge. Checkout https://github.com/whart222/staged-recipes , which is a branch of the conda-forge staged recipes repos. I setup a recipe for pyutilib,which was just accepted. This recipe was quite similar to the recipe you developed for CAChemE, so it should be easy to adapt these recipes and promote their support from a broader community!

[astrojuanlu]

That's great! For reference, this is the relevant PR

https://github.com/conda-forge/staged-recipes/pull/331

In case you are willing to start rolling some feedstocks there, you can find the Linux & OS X versions of our recipes in another branch:

https://github.com/CAChemE/pyomo-recipes/tree/osx-travis

(This is way their process is so much better)

In particular we'll have to work a little bit on the ipopt recipe, because here we're taking the easy route and just packaging the binaries available on the website.

[whart222]

The PyUtilib feedstock was a learning experience, but I think it'll be easy to adapt that recipe for pure-Python packages.

For packages that involve C++ software libraries, I think it might be a lot more work. In particular, I think that native builds of the COIN-OR software will be challenging. I like the idea of continuing to use *_bin packages in the near term. That will help us maintain a core set of "working" packages even as we work on a longer-term solution. Make sense?

--Bill

On Wed, Apr 13, 2016 at 2:36 AM, Juan Luis Cano Rodríguez < notifications@github.com> wrote:

That's great! For reference, this is the relevant PR

conda-forge/staged-recipes#331 https://github.com/conda-forge/staged-recipes/pull/331

In case you are willing to start rolling some feedstocks there, you can find the Linux & OS X versions of our recipes in another branch:

https://github.com/CAChemE/pyomo-recipes/tree/osx-travis

(This is way their process is so much better)

In particular we'll have to work a little bit on the ipopt recipe, because here we're taking the easy route and just packaging the binaries available on the website.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/CAChemE/pyomo-recipes/issues/7#issuecomment-209254041

[astrojuanlu]

You guessed right. I actually have some experience making FEniCS (a finite-element Python & C++ library) to work accross Linux distributions and it can be a real pain. It's fair to go on using *_bin packages in the short term - we will do a huge service to the scientific community if we provide "real" packages but that can come later.

[whart222]

FYI, I've also created a pull request for Pyomo:

conda-forge/staged-recipes#449

This looks close to being adopted. Once that's done, the next step is to work on setting up packages that integrated into pyomo.extras (Pyro, etc). I don't ever foresee conda-forge supporting binary only packages. Should we plan to support that in CAChemE? I think I might convince the COIN-OR folks to do that?

[astrojuanlu]

I see - I understand that they don't want to support binary-only packages, so getting them here would be definitely a good option in my opinion.

We can either remove the packages already present in conda-forge from this repo (pyomo, pyutilib and dependencies) and leave just the solvers or create a new one. In either case, I think we should try to adopt conda-smithy and some other bits of the conda-forge infrastructure because it's much better than what we have here. In particular:

• I want the pull requests to go green if all the tests pass, instead of checking if anaconda upload failed.
• It would be better to have a single branch for all the operative systems (right now the Linux and OS X recipes haven't seen all your updates and improvements).

How does this sound for you and the COIN-OR team? cc @CAChemE/cachemers

[whart222]

For now, I recommend leaving the CAChemE site as is. I think it'll make sense to migrate pyomo and pyomo.extras (and their dependencies) to conda-forge, but I wouldn't break something that's already working. Even so, I agree that it makes sense to adopt their infrastructure.

The COIN-OR folks are interested in supporting compiled versions of their toolset, but that might take a while to get going. However, we haven't really discussed whether this should be hosted on conda-forge. Is there a rationale for NOT using conda-forge?

I haven't had a chance to discuss support for binary-only COIN-OR distributions. I'll work on that with them in late May.

[astrojuanlu]

There isn't really a rationale for not using conda-forge - I just suspect that they will be reluctant to host binary-only packages, but the best we can do is ask them instead of trusting my gut feeling :) I have just asked them if it would be okay: https://gitter.im/conda-forge/conda-forge.github.io?at=5721c1a2e8a4670f2b5d45c5

[astrojuanlu]

Just for the record: Filipe Fernandes chimed in and encouraged us to try and build the solvers from source first and @whart222 agreed.

https://gitter.im/conda-forge/conda-forge.github.io?at=5721f4bee8a4670f2b5d54db

Keep in mind that the binary blob is not forbidden, just a plan B :wink:

[whart222]

I've started setting up conda-forge feedstocks for packages used by pyomo.extras. I'm not confident of what we should do for funcdesigner, derapproximator and openopt. These packages require patches and the openopt.org website is down (which suggests that these packages are not being maintained). Is it worth committing to maintaining these packages? I haven't seen evidence that they are widely used by Pyomo users.

[astrojuanlu]

@whart222 If the original maintainers are not working on them anymore and you feel okay to leave them behind, it would be safer to not work on them I guess. They can be an unexpected burden for us and probably not worth the effort.