yaposib installation

[Midnighter]

So I tried installing yaposib and I've run into a few problems. My system setup is probably somewhat non-standard since I have no administrative access and I use Python virtual environments but I have hunted down most problems, I believe.

(1) The first problem was that pkg-config couldn't find python. I'm fairly sure that's not necessary. Python automatically inserts its own include directory when building an extension.

So I changed line 47 in the setup.py script to:

for lib in ('osi',):

and I still find in the build log:

gcc -pthread -fno-strict-aliasing -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPY_FORMAT_LONG_LONG=I64 -DCpx -DGrb -DXpr -I/home/beber/.local/include/coin -I/home/beber/.local/include/coin -I/home/beber/.local/include/lazylpsolverlibs/ilcplex -I/home/beber/.local/include/coin -I/home/beber/.local/include/lazylpsolverlibs -I/home/beber/.local/include/coin -I/home/beber/.local/include/lazylpsolverlibs -I/bla/python/python-2.7/include/python2.7 -c yaposib/Obj.cpp -o build/temp.linux-x86_64-2.7/yaposib/Obj.o

Or is something else missing then? I work with Python virtual environments most of the time, so I wouldn't like to create .pc files for each of them or not even for the different Python versions. If it is really needed why not use os.path.join(sys.prefix, "include") and os.path.join(sys.prefix, "lib")?

(2) Unfortunately, the build process then fails on:

gcc -pthread -fno-strict-aliasing -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPY_FORMAT_LONG_LONG=I64 -DCpx -DGrb -DXpr -I/home/beber/.local/include/coin -I/home/beber/.local/include/coin -I/home/beber/.local/include/lazylpsolverlibs/ilcplex -I/home/beber/.local/include/coin -I/home/beber/.local/include/lazylpsolverlibs -I/home/beber/.local/include/coin -I/home/beber/.local/include/lazylpsolverlibs -I/package/mariux64/python/python-2.7/include/python2.7 -c yaposib/Problem.cpp -o build/temp.linux-x86_64-2.7/yaposib/Problem.o
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ [enabled by default]
yaposib/Problem.cpp:7:39: fatal error: coin/OsiSolverInterface.hpp: No such file or directory
 #include <coin/OsiSolverInterface.hpp>
                                       ^
compilation terminated.
error: command 'gcc' failed with exit status 1

This seems to be due to the following output of pkg-config:

$ pkg-config --libs --cflags osi
-I/home/beber/.local/include/coin  -L/home/beber/.local/lib -lOsi -lCoinUtils -lbz2 -lz -llapack -lblas -lm

I'm not sure why it would add coin to the include path. Anyways, repeating the build with CFLAGS=-I/home/beber/.local/include python setup.py build works.

(3) So I finally built yaposib in the following way (this avoids setting the LD_LIBRARY_PATH):

CFLAGS="-I/home/beber/.local/include -Wl,-rpath,/home/beber/.local/lib" python setup.py install

When I then run yaposib-config I get the following error:

Traceback (most recent call last):
  File "/home/beber/.virtualenvs/lpfacade2/bin/yaposib-config", line 3, in <module>
    from yaposib.test.test_yaposib import main
  File "/home/beber/.virtualenvs/lpfacade2/lib/python2.7/site-packages/yaposib/__init__.py", line 1, in <module>
    from _yaposib import *
ImportError: /usr/lib/libboost_python.so.1.54.0: undefined symbol: PyUnicodeUCS2_FromEncodedObject

and now I'm really stuck. Boost was installed by the administrator, is it possible that there is a mismatch between the Python version I'm using and the Python that libboost_python.so.1.54.0 was compiled against? I have not yet used Boost:Python so I don't know if that is a problem.

[chmduquesne]

Hi,

Thank you for the long bug report!

I did not give a try to virtualenvs, but obviously I should have.

Anyway, it sounds like there is a mismatch between the python of your virtualenv and the boost library you are using. Most likely the python from your virtualenv has been built using 2-byte Unicode characters, and libboost_python was compiled using a Python with 4-byte Unicode characters (I found the explanation here: http://effbot.org/pyfaq/when-importing-module-x-why-do-i-get-undefined-symbol-pyunicodeucs2.htm).

I will probably need to modify the build system to add a check on sys.maxunicode at build time and check libboostpython for the presence of PyUnicodeUCS2* or PyUnicodeUCS4_* in order to warn the user about incompatibilities.

I'll reply to your other points:

1) Great finding! I'll test this on my home desktop (running archlinux) and if it works, I'll definetely change the build system to remove the pkg-config for python.

2) Yes, coin needs to be included. Not sure why the build system did not pick that by itself.

3) So yeah, it must be an incompatibility between the python of your virtualenv and the python libboost_python was built against. The strange thing is that the python executable used by virtualenv is normally the one of your path. So either there is something weird in your setup, or it is a bug in the libboost_python of your distrib. Please investigate and let me know.

Cheers!

[Midnighter]

2) Yes, coin needs to be included. Not sure why the build system did not pick that by itself.

Well, pkg-config does return -I/home/beber/.local/include/coin but when including in the usual way as you've done #include <coin/header> that fails. So in the .pc file the includedir=${prefix}/include/coin which is, of course, the correct directory for its header files but it is not the way that the headers are normally included. I'm not sure why it's defined that way rather than includedir=${prefix}/include.

3) So yeah, it must be an incompatibility between the python of your virtualenv and the python libboost_python was built against.

We have Python 2.6.8, 2.7.3, 2.7.8, 3.3.0 and 3.4.1 on our system :sweat_smile: so I'd have to install boost into each virtualenv. That's also why I was asking about Cython recently. Using Cython would automate a lot of the work and usually leads to more maintainable C interfaces. Then again, it would have to be written from scratch. I think, I'll check for one of the versions and report back to you.

[Midnighter]

So I tried with Python 2.7.3 which is the one that libboost_python was compiled with and it seems to work. Only remaining problem now is that I installed the lazylpsolverlibs and did not provide the CPLEX or XPRS library paths and hence yaposib.available_solvers() bails with:

lazylpsolverlibs: Library lookup failed! Suggestions:
- point the LAZYLPSOLVERLIBS_CPLEX_LIB environment variable to the full path of the library
- or more secure, symlink /usr/local/lib/libcplex.so to the full path of the library (you need root access).
(process:28331): GModule-CRITICAL **: g_module_symbol: assertion `module != NULL' failed
lazylpsolverlibs: the symbol CPXopenCPLEX could not be found! Exiting your program to avoid a segfault.

[chmduquesne]

All right.

So regarding #issuecomment-56020138:

2) Ah ok, I missed the coin/header thing. Ok, so this is intended: I am including the header the way I am supposed to. Any library that ships headers is supposed to put them in a dedicated directory, to avoid naming conflicts (Imagine projects foo and bar both ship their own matrix.h). It normally ships in the default include path, so that you just need to do

#include <foo/matrix.h>

or

#include <bar/matrix.h>

The only exception is in the standard library. If pkgconfig is returning the wrong thing, then it is a bug in the coin installation process. AFAIK, debian patches coin to get it right.

3) Sorry... I am not against cython or anything, but it is just that I picked what seemed to be the best project that was around at the time.

Now, regarding #issuecomment-56027448:

You're lucky, I am also the author of lazylpsolverlibs (it is not completely finished, but if it works for you, that's cool). Basically, you must do

export LAZYLPSOLVERLIBS_CPLEX_LIB=/path/to/libcplex.so

and

export LAZYLPSOLVERLIBS_XPRS_LIB=/path/to/libxprs.so

The names may vary: libcplex usually has the version in the name. Run the test tool that is bundled with lazylpsolverlibs.

However, should you run into an error, I advise to avoid using lazylpsolverlibs because it is unmaintained: I am not an academic anymore, and thus I can't download these for free, so as a consequence I did not update the project to follow the upgrades. It is probably better to recompile coin completely.

[Midnighter]

3) Sorry... I am not against cython or anything, but it is just that I picked what seemed to be the best project that was around at the time.

No need to apologize, it's great to have this project!

About lazylpsolverlibs: I have Gurobi installed and set the path to it but I wanted to simulate CPLEX not being there and I have no intention of installing libxprs. So I thought I could develop against lazylpsolverlibs and then anyone using that can drop in whatever solver they want to. The current behavior is just different from what I expected. At the moment lazylpsolverlibs exits the program when the solver libs are unavailable. I thought it would throw an error that is then handled making the solver unavailable.

I guess, I will recompile COIN-OSI without lazylibs and then compile with all the solver that I do want in.

[tkralphs]

Hi all,

Sorry to be late to the conversation. I've been using yaposib and have also agreed to help with maintenance going forward. I am also intimately familiar with the inner workings of the COIN build system and generation of the .pc files, so I can comment on a few things for archiving purposes.

First, I found the same issues with building as were reported above, but failed to open issues about them at the time or to push my local mods. Sorry! On OS X, the .pc file for Python is not included with the standard distribution and removing dependence on it does not seem to cause problems.

About issuecomment-56031035, the reason why the .pc files list the include directory as <prefix>/include/coin is more or less historical. There was already a pretty big code base when we switched to using the auto tools for builds and tried to make it possible to install COIN solvers in standard system locations. We decided to install the headers in <prefix>/include/coin for the reasons Christophe-Marie mentioned.

However, many, many source files already had includes that did not have the coin/ prefix. Obviously, we could have changed all of these includes (a bit painful but possible), but a bigger problem was that the way we initially designed the build system (and the way it still is), we build all project first and then install them rather than building and installing them one by one. In order to do this, we link against uninstalled libraries, for which the headers are not inside the coin/ subdirectory yet (this is planned to be changed in a future version of the build tools under development). The easiest solution at the time was to simply have two different .pc files for each project---one for finding the installed headers and one for finding the uninstalled headers.

So there you have it---a bit clunky but it works for now until we find the time to finish the new version of the build system. My solution to the problem was just to remove the coin/ prefix from the includes in Binding.cpp and Problem.cpp, which allows everything to build.