search

coin-or / python-mip

Python-MIP: collection of Python tools for the modeling and solution of Mixed-Integer Linear programs
Eclipse Public License 2.0
525 stars 92 forks source link

Include source code for CBC libraries #70

Open pmla opened 4 years ago

pmla commented 4 years ago

The CBC libraries in the git repository are pre-built. This makes it hard to debug any issues occuring within the library since we don't have the source.

It also makes the git repository very large (it is currently over 200mb). Git is designed with text in mind. It will store a diff of text files, but for binary files it stores a copy of every version. Here is some more info.

Fortunately it is usually not too difficult to build a shared library from source. It can be specified in the setup.py file. If you put the CBC source files somewhere I can try to do it.

jGaboardi commented 4 years ago

@pmla, would this also make #56 more manageable?

h-g-s commented 4 years ago

Hi Peter,

This is an interesting discussion.  Right now we provide binaries so that there is no need of compiling anything when installing Python-MIP. I had some bad experiences where numpy would not compile on my machine depending on the versions of other packages that I have installed (such as blas and lapack if I remember correctly).

Building is easy, but building it statically, so that no additional dependency is need is complicated.

The build script now is in scripts/buildCBCLinux.sh

I'm open to discussions/proposals.

Cheers

Em 3/13/20 12:08 PM, Peter Larsen escreveu:

The CBC libraries in the git repository are pre-built. This makes it hard to debug any issues occuring within the library since we don't have the source.

It also makes the git repository very large (it is currently over 200mb). Git is designed with text in mind. It will store a diff of text files, but for binary files it stores a copy of every version. Here is some more info. https://medium.com/@buzaa/why-you-should-not-push-binary-files-to-git-and-what-to-do-if-you-do-29f34692cef0

Fortunately it is usually not too difficult to build a shared library from source. It can be specified in the |setup.py| file. If you put the CBC source files somewhere I can try to do it.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/coin-or/python-mip/issues/70, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4VZOVLPBJGTFJLGSK43BLRHJEAHANCNFSM4LHEPHVA.

--

Haroldo Gambini Santos Computing Department Universidade Federal de Ouro Preto - UFOP email: haroldo@ufop.edu.br home/research page: www.decom.ufop.br/haroldo

It has long been an axiom of mine that the little things are infinitely the most important. -- Sir Arthur Conan Doyle, "A Case of Identity"

pmla commented 4 years ago

would this also make #56 more manageable?

@jGaboardi, perhaps! I don't know much about conda though.

pmla commented 4 years ago

@h-g-s OK, thanks. It looks quite complex :) I will have a poke around.

tkralphs commented 4 years ago

I just want to mention for completeness that Cbc is open source and hosted here in the COIN-OR Github organization. The topic of building Cbc from source, especially for the purpose of using it with Python modules, is well-explored territory. A lot of effort has been put into making Cbc as easy to build on the major platforms as possible and there are multiple Python packages that wrap it. Building it from source is a bit complex, since it its comprised of multiple independent packages, each of which has to be built separately. However, there is a build script coinbrew which automates the process and makes building easy.

There are also already multiple conda recipes (see here and particularly here), as well as Linux packages and homebrew recipes for OS X. If you want to improve the situation on your favorite platform, the best idea would be to contribute to what already exists rather than starting something new.

As for binaries versus building from source, it seems like a good idea to make it easy for most users to get Cbc in binary form, as most people will not end up wanting to debug it and anyone who wants to build from source and debug can easily do so. But it is indeed not a good idea to host binaries in git. It seems better for the setup script to just download the binaries from bintray or for the user to be asked to install Cbc themselves.

pgoelz commented 4 years ago

To add to the conversation, we have been chasing a bug where CBC silently crashes and takes the entire program with it. The bug only appears on Windows and is not triggered by any specific ILP. Our usage case optimizes a fixed sequence of ILPs and will crash at a different ILP each time (I have verified that the ILPs are indeed the same). The cause seems to be an access violation within the call to cbclib.Cbc_solve() in cbc-c-windows-x86-64.dll, but I don’t know how to debug further. I was hoping to recompile CBC with debug flags, but coinbrew fails with checking how to get verbose linking output from ... configure: WARNING: compilation failed. I also tried using your compilation scripts (./downloadCBC.sh, then ./buildCBCWin.sh, on a linux machine), but got the following error output:

...
checking whether to build static libraries... yes
checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/home/ec2-user/python-mip/scripts/Cbc':
configure: error: C++ preprocessor "/lib/cpp" fails sanity check 
See `config.log' for more details
make: *** No targets specified and no makefile found.  Stop.
make: *** No rule to make target `install'.  Stop.

I’m attaching config.log. Am I using the scripts wrong? So far, I have not modified any flags.

esieke commented 4 years ago

Hello Everyone, in a project i try to use MIP on an aarch64 platform. That means that I have to compile the CBC package each time. The biggest disadvantage for me is that MIP isn't related to an official CBC release. If I use downloadCBC.sh, the code base can change from day to day. Or did I miss something?

Suggestion: Use cowbrew in downloadCBC.sh to fetch the source code related to a stable CBC release. e.g. coinbrew fetch Cbc:stable/2.10

UPDATE Sorry I didn't see issue #35! The questions are more or less the same!

pmla commented 4 years ago

Thanks for all the feedback. A quick summary:

  1. We are duplicating the efforts of the Cbc project by building our own binaries.
  2. We do not support all platforms.
  3. The binaries are not tied to an official release. If a user does some debugging and finds a problem, it is quite likely that we all have different Cbc versions. See issue #35.
  4. The git repository is very large because git does not handle binary files well, and the binaries are replaced frequently.
  5. The binaries are difficult to debug.

One solution would be to fetch the binaries from bintray as @tkralphs suggests. Unfortunately PyPI / pip no longer allows external files to be downloaded during installation, as this feature was often misused by malicious packages.

Alternatively we could replace the binaries in python-mip/mip/libraries/ with the binaries built by the Cbc project (for all platforms). This would solve issues 1-3. Since the official releases do not come too frequently this would also help with issue 4. Issue #35 is the obstacle to this.

We could also consider building everything on the user's machine, by switching from setuptools to scikit-build. This summary of distribution tools is helpful. This proposal is probably unworkable, as building on Windows is very difficult.

I think solving #35 is the way to go. Opinions?

tkralphs commented 4 years ago

The biggest problem with tying binaries to a release is just that it is only recently, with the new version of the BuildTools, that we can easily build DLLs with the Visual Studio compiler on Windows with coinbrew in an automated fashion. This can't really be done easily with any existing releases. We're working on getting a new stable version and new releases out from the current master, but there is still some work to be done there. In the meantime, we can at least tie releases to a particular SHA. Once we are rolling with releases from a new stable version based on the current master version and things stabilize, then I don't think there will be much question but that just installing the latest release will be the right thing to do. Users shouldn't be doing anything with master in normal times. So we need a bit of patience to pull this all together.

There is a solution for one part that is mentioned in #35---building a single DLL using coinbrew---but I'm not sure we quite got all the pieces of the puzzle in place. @h-g-s did you ever get all the changes committed for this? It seemed like all that was needed was to remove some code from header files.

The best solution overall is probably going to be different on different platforms.

I think there is a path to make all of this work smoothly on all platforms and we've already been slowly working towards that in support of the many projects that use Cbc. If anyone wants to contribute to filling in any of the missing pieces, that would be great. Here are things that could help.

Let me know if you want to contribute! I guess I should advertise some of this to the Cbc community at large as well.

tkralphs commented 4 years ago

@pgoelz Please open a separate issue and provide more details. What compiler are you trying to build with?

tuliotoffolo commented 4 years ago

Hi @pmla, just wanted to mention that we want and eventually will remove the binaries from the repo. However, we must first find the best way of adding them to the pip package, since we must keep Python-MIP straightforward to install in most windows, linux and mac machines. We are currently looking into that.