search

coin-or / python-mip

Python-MIP: collection of Python tools for the modeling and solution of Mixed-Integer Linear programs
Eclipse Public License 2.0
530 stars 92 forks source link

Fails to work on Raspberry Pi #19

Closed kokil01 closed 4 years ago

kokil01 commented 4 years ago

Hi

I am trying to use the MIP package on raspian buster but it fails with the error 

Osi_getNumCols = cbclib.Osi_getNumCols
NameError: name 'cbclib' is not defined

Which is expected since only linux, windows and mac osx binaries are included by default.

However cbc is available in the debian repositories and can be installed using

sudo apt-get install coinor-cbc

which will place cbc binary in path linking to following so's

        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76fa9000)
        libCbcSolver.so.3 => /usr/lib/arm-linux-gnueabihf/libCbcSolver.so.3 (0x76ecc000)
        libCbc.so.3 => /usr/lib/arm-linux-gnueabihf/libCbc.so.3 (0x76dd4000)
        libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x76daa000)
        librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0x76d93000)
        libCgl.so.1 => /usr/lib/arm-linux-gnueabihf/libCgl.so.1 (0x76c9a000)
        libOsiClp.so.1 => /usr/lib/arm-linux-gnueabihf/libOsiClp.so.1 (0x76c4c000)
        libClpSolver.so.1 => /usr/lib/arm-linux-gnueabihf/libClpSolver.so.1 (0x76bec000)
        libClp.so.1 => /usr/lib/arm-linux-gnueabihf/libClp.so.1 (0x76a7a000)
        libOsi.so.1 => /usr/lib/arm-linux-gnueabihf/libOsi.so.1 (0x76a24000)
        libCoinUtils.so.3 => /usr/lib/arm-linux-gnueabihf/libCoinUtils.so.3 (0x7690c000)
        libbz2.so.1.0 => /lib/arm-linux-gnueabihf/libbz2.so.1.0 (0x768ec000)
        libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0x768c1000)
        liblapack.so.3 => /usr/lib/arm-linux-gnueabihf/liblapack.so.3 (0x76376000)
        libblas.so.3 => /usr/lib/arm-linux-gnueabihf/libblas.so.3 (0x76310000)
        libstdc++.so.6 => /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 (0x761c9000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76147000)
        libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x7611a000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x75fcc000)
        /lib/ld-linux-armhf.so.3 (0x76fbe000)
        libgfortran.so.5 => /usr/lib/arm-linux-gnueabihf/libgfortran.so.5 (0x75ef3000)

Is it possible to use these to get MIP working on buster?

Thanks!

h-g-s commented 4 years ago

Hello,

Python-MIP now uses the development version of cbc (cbc trunk) built with some customized options. I'll would love to see it running in the Rabsberry pi (planning to buy one in the future). I'll provide the instructions for you to build it. If you provide me the resulting binary I can include in the Python-MIP package.

First, download the development version of cbc:

svn checkout https://projects.coin-or.org/svn/Cbc/trunk cbctrunk

cd cbctrunk

cd ThirdParty/Lapack/

./get.Lapack

cd ../Blas/

./get.Blas

cd ../../

cp ~/dev/mip/scripts/buildCBCLinux.sh ./

(it will probably be better to modify the make -j 2 line of this script to make -j 1 since  the raspberry doesn't have too much memory and building with link time optimization  flto uses a lot of memory)

bash buildCBCLinux.sh

(you'll need gcc and gfortran)

Cheers,

Haroldo

On 13/10/2019 03:49, kokil01 wrote:

Hi

I am trying to use the MIP package on raspian buster but it fails with the error

|Osi_getNumCols = cbclib.Osi_getNumCols NameError: name 'cbclib' is not defined |

Which is expected since only linux, windows and mac osx binaries are included by default.

However cbc is available in the debian repositories and can be installed using

|sudo apt-get install coinor-cbc |

which will place cbc binary in path linking to following so's

|/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76fa9000) libCbcSolver.so.3 => /usr/lib/arm-linux-gnueabihf/libCbcSolver.so.3 (0x76ecc000) libCbc.so.3 => /usr/lib/arm-linux-gnueabihf/libCbc.so.3 (0x76dd4000) libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x76daa000) librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0x76d93000) libCgl.so.1 => /usr/lib/arm-linux-gnueabihf/libCgl.so.1 (0x76c9a000) libOsiClp.so.1 => /usr/lib/arm-linux-gnueabihf/libOsiClp.so.1 (0x76c4c000) libClpSolver.so.1 => /usr/lib/arm-linux-gnueabihf/libClpSolver.so.1 (0x76bec000) libClp.so.1 => /usr/lib/arm-linux-gnueabihf/libClp.so.1 (0x76a7a000) libOsi.so.1 => /usr/lib/arm-linux-gnueabihf/libOsi.so.1 (0x76a24000) libCoinUtils.so.3 => /usr/lib/arm-linux-gnueabihf/libCoinUtils.so.3 (0x7690c000) libbz2.so.1.0 => /lib/arm-linux-gnueabihf/libbz2.so.1.0 (0x768ec000) libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0x768c1000) liblapack.so.3 => /usr/lib/arm-linux-gnueabihf/liblapack.so.3 (0x76376000) libblas.so.3 => /usr/lib/arm-linux-gnueabihf/libblas.so.3 (0x76310000) libstdc++.so.6 => /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 (0x761c9000) libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76147000) libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x7611a000) libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x75fcc000) /lib/ld-linux-armhf.so.3 (0x76fbe000) libgfortran.so.5 => /usr/lib/arm-linux-gnueabihf/libgfortran.so.5 (0x75ef3000) |

Is it possible to use these to get MIP working on buster?

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/coin-or/python-mip/issues/19?email_source=notifications&email_token=AB4VZOWC6MNAC4F33TMZYLTQOLAHFA5CNFSM4JAFX5LKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HRNIWCQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4VZOTLU3AQDWBTCOQD2LDQOLAHFANCNFSM4JAFX5LA.

--

Haroldo Gambini Santos Computing Department Universidade Federal de Ouro Preto - UFOP email: haroldo@ufop.edu.br home/research page: www.decom.ufop.br/haroldo

It has long been an axiom of mine that the little things are infinitely the most important. -- Sir Arthur Conan Doyle, "A Case of Identity"

kokil01 commented 4 years ago

Thanks for providing the instructions.

I tried them and it failed because it could not link to libquadmath. Looks like its not supported on arm platforms yet - https://packages.debian.org/buster/libquadmath0

I modified the buildCBCLinux.sh locally by passing --disable-libquadmath option to the configure script. Lets see if it works.

Update: Unfortunately it didn't work.

kokil01 commented 4 years ago

Hi

Spend sometime looking into this. I had to make 2 changes and I was able to compile. Since libquadmath is not available on Pi I had to remove it from linking. Here is the final bash script

if [ ! -d "Cbc" ];
then
        echo "call this script in the CBC source directory"
        exit
fi
if [ ! -f "configure" ];
then
        echo "call this script in the CBC source directory"
        exit
fi

export CFLAGS="-Ofast -fPIC -flto -DNDEBUG -fprefetch-loop-arrays -I/opt/gcc/include/"
export FFLAGS="-Ofast -fPIC -flto -DNDEBUG -I/opt/gcc/include/"
export CXXFLAGS="-Ofast -fPIC -flto -DNDEBUG -I/opt/gcc/include/"
export LDFLAGS="-Ofast -fPIC -L/opt/gcc/lib -flto -static-libgcc -static-libstdc++ -static-libgfortran"
./configure --enable-cbc-parallel --enable-static --disable-shared --disable-libquadmath
make clean
make -j 1
g++ -shared -Ofast -fPIC -o cbc-c-linux-x86-64.so \
-I~/prog/include/ -I./CoinUtils/src/ -I./Osi/src/ -I./Clp/src/ -I./Clp/src/ \
-I./Cgl/src/ -I./Cbc/src/ -I./Osi/src/Osi/ -I./Clp/src/OsiClp/ -DCBC_THREAD \
./Cbc/src/Cbc_C_Interface.cpp -L/opt/gcc/lib64/ -L./Cbc/src/.libs/ -L./Cgl/src/.libs/ \
-L./Cgl/src/.libs/ -L./Osi/src/Osi/.libs/ -L./Clp/src/OsiClp/.libs/ -L./Clp/src/.libs/ \
-L./CoinUtils/src/.libs/ -L./Cgl/src/.libs/ -L./Clp/src/OsiClp/.libs/ -L./Clp/src/.libs/ \
-L./ThirdParty/Lapack/.libs/ -L./ThirdParty/Blas/.libs/ \
 -lCbcSolver -lCbc -lpthread -lrt -lCgl -lOsiClp -lClpSolver -lClp -lOsi -lCoinUtils \
 -lcoinlapack -lcoinblas -lgfortran -lm -static-libgcc -static-libstdc++ -static-libgfortran

I have attached the produced binary file.

But I think it still does not work. Because when I do a = ffi.dlopen("/tmp/cbc-c-linux-x86-64.so") with the generated file it produces the following error - 

OSError: cannot load library '/tmp/cbc-c-linux-x86-64.so': /tmp/cbc-c-linux-x86-64.so: undefined symbol: BZ2_bzWriteClose.  Additionally, ctypes.util.find_library() did not manage to locate a library called '/tmp/cbc-c-linux-x86-64.so'

cbc-c-linux-x86-64.so.zip

kokil01 commented 4 years ago

Looking for all undefined symbols looks like there are a lot of them

nm /tmp/cbc-c-linux-x86-64.so | grep 'U '

yields

        U BZ2_bzRead
         U BZ2_bzReadClose
         U BZ2_bzReadOpen
         U BZ2_bzWrite
         U BZ2_bzWriteClose
         U BZ2_bzWriteOpen
         U _gfortran_stop_string@@GFORTRAN_8
         U _gfortran_string_len_trim@@GFORTRAN_8
         U _gfortran_st_write_done@@GFORTRAN_8
         U _gfortran_st_write@@GFORTRAN_8
         U _gfortran_transfer_character_write@@GFORTRAN_8
         U _gfortran_transfer_integer_write@@GFORTRAN_8
         U gzclose
         U gzopen
         U gzread
         U gzwrite
kokil01 commented 4 years ago

It worked!!! Just needed to add bz2 as a library

g++ -shared -Ofast -fPIC -o cbc-c-linux-x86-64.so \
                                        -I~/prog/include/ -I./CoinUtils/src/ -I./Osi/src/ -I./Clp/src/ -I./Clp/src/ \
                                        -I./Cgl/src/ -I./Cbc/src/ -I./Osi/src/Osi/ -I./Clp/src/OsiClp/ -DCBC_THREAD \
                                        ./Cbc/src/Cbc_C_Interface.cpp -L/opt/gcc/lib64/ -L./Cbc/src/.libs/ -L./Cgl/src/.libs/ \
                                        -L./Cgl/src/.libs/ -L./Osi/src/Osi/.libs/ -L./Clp/src/OsiClp/.libs/ -L./Clp/src/.libs/ \
                                        -L./CoinUtils/src/.libs/ -L./Cgl/src/.libs/ -L./Clp/src/OsiClp/.libs/ -L./Clp/src/.libs/ \
                                        -L./ThirdParty/Lapack/.libs/ -L./ThirdParty/Blas/.libs/ \
                                         -lCbcSolver -lCbc -lpthread -lrt -lCgl -lOsiClp -lClpSolver -lClp -lOsi -lCoinUtils -lbz2 \
                                         -lcoinlapack -lcoinblas -lgfortran -lm -static-libgcc -static-libstdc++ -static-libgfortran

Here is the .so file which loads in python. cbc-c-linux-x86-64.so.zip

h-g-s commented 4 years ago

Great ! Now the cbc binary should be renamed to reflect the achitecture of the build. Could you enter an lscpu command and check which architecture is reported ? Yours is a raspberry 3 or 4 ?

After, tests in cbc.py should include this new architecture, the current code is:

(cbc.py line 23) os_is_64_bit = maxsize > 2**32 INF = float('inf') cut_idx = 0

for variables and rows

MAX_NAME_SIZE = 512

DEF_PUMPP = 30

try: pathmip = dirname(mip.file) pathlib = os.path.join(pathmip, 'libraries') libfile = '' if 'linux' in platform.lower(): if os_is_64_bit: libfile = os.path.join(pathlib, 'cbc-c-linux-x86-64.so') elif platform.lower().startswith('win'): if os_is_64_bit: libfile = os.path.join(pathlib, 'cbc-c-windows-x86-64.dll') else: libfile = os.path.join(pathlib, 'cbc-c-windows-x86-32.dll') elif platform.lower().startswith('darwin') or platform.lower().startswith('macos'): if os_is_64_bit: libfile = os.path.join(pathlib, 'cbc-c-darwin-x86-64.dylib')

--

Haroldo Gambini Santos Computing Department Universidade Federal de Ouro Preto - UFOP email: haroldo@ufop.edu.br Haroldo.GambiniSantos@cs.kuleuven.be home/research page: www.decom.ufop.br/haroldo

It has long been an axiom of mine that the little things are infinitely the most important. -- Sir Arthur Conan Doyle, "A Case of Identity"

On Sat, 2 Nov 2019, kokil01 wrote:

It worked!!! Just needed to add bz2 as a library

g++ -shared -Ofast -fPIC -o cbc-c-linux-x86-64.so \ -I~/prog/include/ -I./CoinUtils/src/ -I./Osi/src/ -I./Clp/sr c/ -I./Clp/src/ \ -I./Cgl/src/ -I./Cbc/src/ -I./Osi/src/Osi/ -I./Clp/src/OsiCl p/ -DCBC_THREAD \ ./Cbc/src/Cbc_C_Interface.cpp -L/opt/gcc/lib64/ -L./Cbc/src/ .libs/ -L./Cgl/src/.libs/ \ -L./Cgl/src/.libs/ -L./Osi/src/Osi/.libs/ -L./Clp/src/OsiClp /.libs/ -L./Clp/src/.libs/ \ -L./CoinUtils/src/.libs/ -L./Cgl/src/.libs/ -L./Clp/src/OsiC lp/.libs/ -L./Clp/src/.libs/ \ -L./ThirdParty/Lapack/.libs/ -L./ThirdParty/Blas/.libs/ \ -lCbcSolver -lCbc -lpthread -lrt -lCgl -lOsiClp -lClpSolver -lClp -lOsi -lCoinUtils -lbz2 \ -lcoinlapack -lcoinblas -lgfortran -lm -static-libgcc -stat ic-libstdc++ -static-libgfortran

Here is the .so file which loads in python. cbc-c-linux-x86-64.so.zip

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, orunsubscribe.[AB4VZOQOISGIXPX7X24P4ULQRWB7NA5CNFSM4JAFX5LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJK TDN5WW2ZLOORPWSZGOEC44SWI.gif]

h-g-s commented 4 years ago

just added this to the TODO list of the project, I'll remove as an issue for now

corneel27 commented 2 years ago

I was installing mip on a raspberry pi, but it is still not working "out of the box" I have build the binary by myself from https://github.com/coin-or / Cbc And with a softlink to the builded binary the mip library works fine. But I want to use it in Appdaemon of Home Assistant and that docker image doesn't allow me to install binaries. Is it possible to get this to work with an arm-binary?