Trying to link mip to Debian's CBC package throws undefined error

[sjferc]

I installed coinor-cbc to my armhf system, and did this:

export PMIP_CBC_LIBRARY="/usr/lib/arm-linux-gnueabihf/libCbcSolver.so.3.9.9"
export LD_LIBRARY_PATH="/usr/lib/arm-linux-gnueabihf/"

as established here.

Then, when I try to execute it on Python's console, I do:

>>> from mip import Model, MAXIMIZE, CBC, INTEGER, OptimizationStatus
>>> model = Model(sense=MAXIMIZE, solver_name=CBC)

And I inmediately get the error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/mip/model.py", line 93, in __init__
    import mip.cbc
  File "/usr/local/lib/python3.7/dist-packages/mip/cbc.py", line 600, in <module>
    Osi_getNumCols = cbclib.Osi_getNumCols
  File "/usr/local/lib/python3.7/dist-packages/cffi/api.py", line 912, in __getattr__
    make_accessor(name)
  File "/usr/local/lib/python3.7/dist-packages/cffi/api.py", line 908, in make_accessor
    accessors[name](name)
  File "/usr/local/lib/python3.7/dist-packages/cffi/api.py", line 838, in accessor_function
    value = backendlib.load_function(BType, name)
AttributeError: function/symbol 'Osi_getNumCols' not found in library '/usr/lib/arm-linux-gnueabihf/libCbcSolver.so.3.9.9': /usr/lib/arm-linux-gnueabihf/libCbcSolver.so.3.9.9: undefined symbol: Osi_getNumCols

I am using

• Python 3.7.3
• mip 1.12.0
• coinor-cbc 2.9.9+repack1-1
• Debian GNU/Linux 10 (buster)
• Kernel: Linux 4.14.78-g8e54a4b
• Architecture: armv7l

[h-g-s]

Hi,

I think that Debian probably includes the last stable (API stable) version of CBC.

Python-MIP requires the development version of CBC.

From: sjferc notifications@github.com Reply-To: coin-or/python-mip reply@reply.github.com Date: Monday, November 30, 2020 at 6:38 AM To: coin-or/python-mip python-mip@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [coin-or/python-mip] Trying to link mip to Debian's CBC package throws undefined error (#165)

I installed coinor-cbc to my armhf system, and did this: export PMIP_CBC_LIBRARY="/usr/lib/arm-linux-gnueabihf/libCbcSolver.so.3.9.9" export LD_LIBRARY_PATH="/usr/lib/arm-linux-gnueabihf/" as established here.

Then, when I try to execute it on Python's console, I do:

from mip import Model, MAXIMIZE, CBC, INTEGER, OptimizationStatus model = Model(sense=MAXIMIZE, solver_name=CBC) And I inmediately get the error: Traceback (most recent call last):   File "", line 1, in   File "/usr/local/lib/python3.7/dist-packages/mip/model.py", line 93, in init     import mip.cbc   File "/usr/local/lib/python3.7/dist-packages/mip/cbc.py", line 600, in     Osi_getNumCols = cbclib.Osi_getNumCols   File "/usr/local/lib/python3.7/dist-packages/cffi/api.py", line 912, in getattr     make_accessor(name)   File "/usr/local/lib/python3.7/dist-packages/cffi/api.py", line 908, in make_accessor     accessorsname   File "/usr/local/lib/python3.7/dist-packages/cffi/api.py", line 838, in accessor_function     value = backendlib.load_function(BType, name) AttributeError: function/symbol 'Osi_getNumCols' not found in library '/usr/lib/arm-linux-gnueabihf/libCbcSolver.so.3.9.9': /usr/lib/arm-linux-gnueabihf/libCbcSolver.so.3.9.9: undefined symbol: Osi_getNumCols I am using Python 3.7.3 mip 1.12.0 coinor-cbc 2.9.9+repack1-1 Debian GNU/Linux 10 (buster) Kernel: Linux 4.14.78-g8e54a4b Architecture: armv7l — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[sjferc]

Okay, so I did install it with coinbrew like this:

export CFLAGS="-Ofast -fPIC -flto -DNDEBUG -fprefetch-loop-arrays -I/opt/gcc/include/"
export FFLAGS="-Ofast -fPIC -flto -DNDEBUG -I/opt/gcc/include/"
export CXXFLAGS="-Ofast -fPIC -flto -DNDEBUG -I/opt/gcc/include/"
export LDFLAGS="-Ofast -fPIC -L/opt/gcc/lib -flto -static-libgcc -static-libstdc++ -static-libgfortran"

coinbrew build Cbc -j 1 --no-third-party --tests none

Which installed latest release, 2.10.5, and properly linked the libraries again but got:

>>> from mip import Model, MAXIMIZE, CBC, INTEGER, OptimizationStatus
>>> model = Model(sense=MAXIMIZE, solver_name=CBC)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/mip/model.py", line 93, in __init__
    import mip.cbc
  File "/usr/local/lib/python3.7/dist-packages/mip/cbc.py", line 603, in <module>
    Osi_getIntegerTolerance = cbclib.Osi_getIntegerTolerance
  File "/usr/local/lib/python3.7/dist-packages/cffi/api.py", line 912, in __getattr__
    make_accessor(name)
  File "/usr/local/lib/python3.7/dist-packages/cffi/api.py", line 908, in make_accessor
    accessors[name](name)
  File "/usr/local/lib/python3.7/dist-packages/cffi/api.py", line 838, in accessor_function
    value = backendlib.load_function(BType, name)
AttributeError: function/symbol 'Osi_getIntegerTolerance' not found in library '/downloads/dist/lib/libCbcSolver.so': /downloads/dist/lib/libCbcSolver.so: undefined symbol: Osi_getIntegerTolerance

Is this still an incorrect version of CBC?

[tkralphs]

You need to do

coinbrew build Cbc@master -j 1 --no-third-party --tests none

to get the development version, but you should remove the build and dist directories first.

[sjferc]

Thank you! I will test that out. Another thing I noticed... from the flags, there is -I/opt/gcc/include/, but actually that directory does not exist locally. Do you know how could I check where is my equivalent?

It also happens with: -L/opt/gcc/lib.

[tkralphs]

In general, you shouldn't need to set any flags manually like you have. I assumed you had some specific reason for doing that and didn't look too closely at them. You usually would not need any flag such as -I/opt/gcc/include/ except in very specific circumstances.

If you want to build static executables, which it looks like you are trying to do, just add --static to coinbrew. Generally speaking, appropriate flags are set during configuration for each project. If you try to set flags on your own, there's a good chance you will mess something up by, e.g., missing out a flag which is necessary for defining an important symbol.

If necessary, you can augment the pre-defined flags using ADD_CXXFLAGS, etc.

[gewesp]

coinbrew build Cbc@master -j 1 --no-third-party --tests none

I tried this today and it worked, however it seems that the library has been renamed in the Cbc master and Python-MIP has not been updated (yet).

To make it work with Python-MIP you need to do the following after the coinbrew command:

cd dist/lib
ln -s libCbc.dylib -> libCbcSolver.dylib

This is for MacOS, for Linux probably replace .dylib by .so.

Using this command, I got Python-MIP working on a Mac M1 ('Apple silicon').

In the long run, hopefully Python-MIP will distribute appropriate binaries and 'freeze' the Cbc version it is using.

EDIT Mar 9, 2022: It also works to point PMIP_CBC_LIBRARY directly to the libCbc dynamic library (as opposed to libCbcSolver), e.g.

export PMIP_CBC_LIBRARY="$CBC_LIBRARIES_DIR/libCbc.dylib"

(Define CBC_LIBRARYES_DIR appropriately).

[tkralphs]

I tried this today and it worked, however it seems that the library has been renamed in the Cbc master and Python-MIP has not been updated (yet).

Yes, there used to be a libCbc and a libCbcSolver, but this separation did not serve much purpose, so the libraries were merged as part of the on-going refactoring being undertaken in the master branch. I'm not sure why you need to do the dynamic linking , though. You should be able to just set PMIP_CBC_LIBRARY to point directly to libCbc, unless I'm missing something.

We are still trying to get to a formal release of the code in Cbc master branch, but it still needs some work and developer bandwidth is severely limited. The work to be done doesn't need that much knowledge of Cbc internals, volunteers are welcome. See coin-or/Cbc/discussions/465.

[gewesp]

I'm not sure why you need to do the dynamic linking , though. You should be able to just set PMIP_CBC_LIBRARY to point directly to libCbc, unless I'm missing something.

Much appreciated Ted! I was following the Python-MIP documentation which mentions dynamic linking:

"Python-MIP uses the CbcSolver shared library [...] In [...] MacOS the extension should be .dll and .dylp, respectively. To force Python-MIP to use your [...] CBC binaries, you can set the PMIP_CBC_LIBRARY environment variable, indicating the full path to this shared library." (emphasis mine)

[tkralphs]

Sorry, I misspoke. I didn't mean to say dynamic linking, I meant symbolic linking. You created a symbolic link from libCbc to libCbcSolver when I think you could just directly point python-mip to libCbc by setting PMIP_CBC_LIBRARY.

[gewesp]

Indeed, that works as well, thanks!

[jurasofish]

Leaving this here for reference

I've just gotten an M1 Mac and I found that using CBC built from today's master failed some tests in my application - mostly just extreme edge cases, so I guess it's just normal variance between different versions of CBC. But I wanted to have CBC as identical as possible to what's in python-mip v1.13.0

So I wanted to build CBC for M1 Mac using the exact same version of CBC that's used in python-mip 1.13.0. The python-mip CBC binary was last updated around 2020-11-15 which seems likely to be built from this commit of CBC. That is, CBC master @ 2020-11-15T22:40:00Z

When building CBC master version using coinbrew it seems that coinbrew checks out master versions of all the dependencies, if I'm interpreting the contents of the .coin-or folder correctly

So, I've put together a little script that checks out all the dependent repositories to master at the time that I want and then builds CBC. And it seems to work, just need env var PMIP_CBC_LIBRARY=/...../libCbcSolver.dylib @tkralphs does this seem kosher to you?

# see https://stackoverflow.com/a/6990682 for git checkout to specific date
bash coinbrew fetch Cbc@master
cd Cbc
git checkout `git rev-list -n 1 --before="2020-11-15T22:41:00Z" --date=iso master`
cd ../Cgl
git checkout `git rev-list -n 1 --before="2020-11-15T22:41:00Z" --date=iso master`
cd ../Clp
git checkout `git rev-list -n 1 --before="2020-11-15T22:41:00Z" --date=iso master`
cd ../CoinUtils
git checkout `git rev-list -n 1 --before="2020-11-15T22:41:00Z" --date=iso master`
cd ../Osi
git checkout `git rev-list -n 1 --before="2020-11-15T22:41:00Z" --date=iso master`
cd ../ThirdParty/ASL
git checkout `git rev-list -n 1 --before="2020-11-15T22:41:00Z" --date=iso master`
cd ../Glpk
git checkout `git rev-list -n 1 --before="2020-11-15T22:41:00Z" --date=iso master`
# mumps doesn't seem to have a master branch
# cd ../Mumps
# git checkout `git rev-list -n 1 --before="2020-11-15T22:41:00Z" --date=iso master`
cd ../..
bash coinbrew build Cbc -j 1 --no-third-party --tests none

[tkralphs]

Yes, in fact, coinbrew has an option for doing exactly what you're trying to do. From the help:

--time check out project and all dependencies at a time stamp

It's admittedly not been used (by me) since when I actually needed it and added it as an option. If you have a chance to try it, that would be great.