Compiling python-igraph from local C igraph installation in anaconda environment

[AlexisPister]

Hi,

I am trying to compile python-igraph from source, using a locally built core C igraph in an anaconda environment, but at the end of the pipeline I have the following error :

import igraph Traceback (most recent call last): File "", line 1, in File "/home/alexis/anaconda3/envs/igraph-test/lib/python3.8/site-packages/igraph-0.9.9-py3.8-linux-x86_64.egg/igraph/init.py", line 26, in from igraph._igraph import ( ImportError: /home/alexis/anaconda3/envs/igraph-test/lib/python3.8/site-packages/igraph-0.9.9-py3.8-linux-x86_64.egg/igraph/_igraph.cpython-386_64-linux-gnu.so: undefined symbol: GOMP_critical_end

Here is what I did :

For the conda env installation : conda install -c conda-forge cmake gcc gxx libxml2=2.9.11 bison flex gmp glpk arpack pkg-config gxx_linux-64

Then I built the core C igraph this way, from the build foldet : cmake .. -DCMAKE_INSTALL_PREFIX=~/anaconda3/envs/igraph-test make make install

Then I built the python igraph library following the instructions from the igraph website (https://igraph.org/python/) : pip download --no-deps --no-binary :all: igraph tar -xvvzf igraph-0.9.9.tar.gz cd igraph-0.9.9 python setup.py build --use-pkg-config python setup.py install

However, at the end I have the import error I pasted above

Anyone have an idea ? I have been stuck on this for 3 days. Thanks

[vtraag]

This is most related to OpenMP being enabled. I'm not sure why it would trigger this error though, I'd need to try to reproduce the problem.

In the meantime, if you configure cmake using IGRAPH_OPENMP_SUPPORT=OFF it should build without OpenMP support. That should resolve the issue for now. If not, let us know.

[vtraag]

I can confirm I was able to reproduce the problem, but only because pkg-config failed to locate the manually installed version of igraph.

To be sure, what I did was the following. First, I create and activate the environment.

conda create -n test -c conda-forge cmake gcc gxx libxml2=2.9.11 bison flex gmp glpk arpack pkg-config gxx_linux-64 pip
conda activate test

Now build and install igraph C core manually:

wget https://github.com/igraph/igraph/releases/download/0.9.6/igraph-0.9.6.tar.gz
tar xvf igraph-0.9.6.tar.gz
cd igraph-0.9.6/
mkdir build
cd build/
cmake .. -DCMAKE_INSTALL_PREFIX=~/anaconda3/envs/test/ -DBUILD_SHARED_LIBS=ON

Note that this builds a a shared library. This is the default being used when using --use-pkg-config to install python-igraph.

Now we go back up (cd ../..) and install python-igraph

pip download --no-deps --no-binary :all: igraph
tar -xvvzf igraph-0.9.9.tar.gz
cd igraph-0.9.9

Now first check what pkg-config gives

pkg-config igraph --cflags --libs

If this doesn't yield include and library paths that start with ~/anaconda3/envs/test, using pkg-config will link to some other library you may have installed on your computer. You can check the PKG_CONFIG_PATH environment variable to see what path it is using. If necessary, you can update it to make sure it correctly picks up the igraph library installed in ~/anaconda3/envs/test by doing

export PKG_CONFIG_PATH=~/anaconda3/envs/test/lib/pkgconfig/

Now, finally, doing

python setup.py install --use-pkg-config

will install python-igraph correctly, and this runs without any issues in this environment.

So, there are a few things to look out for:

• Make sure you compile and install the igraph library in the same conda environment, otherwise things will be messed up with (dynamic) linking across possibly different versions of libraries.
• Make sure you install python-igraph from within the same conda environment.
• Make sure it links to the correct library that you manually built.

If this doesn't work for you still, let us know.

[vtraag]

By the way, it is a bit odd that the conda installed pkg-config doesn't come with a default PKG_CONFIG_PATH that is already properly referring to the path of the current conda environment. Without that, any package relying on pkg-config within a conda environment will run into the problem that it will pick up libraries from outside that conda environment, which is not the idea. Possibly, an issue could be opened for this somewhere upstream.

[szhorvat]

export PKG_CONFIG_PATH=~/anaconda3/envs/test/lib/pkgconfig/

Small note: Many tools have problems with ~ in paths. While this specific case might work, it's safer to use $HOME/foo instead of ~/foo. Especially with CMake I had problems with this. Does it actually work for you @vtraag ?

---

@AlexisPister Can you explain why you are trying to build the C core yourself? Normally, the C core build process is done as part of compiling python-igraph.

[vtraag]

Small note: Many tools have problems with ~ in paths. While this specific case might work, it's safer to use $HOME/foo instead of ~/foo. Especially with CMake I had problems with this. Does it actually work for you @vtraag ?

This does work for me here, but yes, I agree using $HOME would probably be more robust.

@AlexisPister Can you explain why you are trying to build the C core yourself? Normally, the C core build process is done as part of compiling python-igraph.

I can't answer for @AlexisPister of course, but for me, it is useful to link to a separate dynamic library of igraph that I can then easily develop and link to from the Python interface, so that I don't have to re-compile/re-install the Python interface all the time.

[szhorvat]

Sure, there are many legitimate reasons. I am just trying to make sure that @AlexisPister is not wasting his time trying to do this to achieve something that could be done in an easier way.

[AlexisPister]

@vtraag Ok so I followed your post by starting from scratch with a new anaconda environment and it worked ! However during my previous attempts I had checked and pkg-config was actually giving me the good files paths located in the anaconda environement folder.

The two things you did differently was adding pip in the conda install command, and that you added the flag -DBUILD_SHARED_LIBS=ON in the cmake command. So I would say one of those two things fixed my problem ?

Also this time I only did python setup.py install --use-pkg-config instead of python setup.py build --use-pkg-config python setup.py install

But I don't think it makes a difference.

Anyway, thanks a lot for your rapid help !

@szhorvat I had to change the sugiyama layout algorithm to add an optimization, and the algorithm is written in the core of igraph

[vtraag]

It's difficult to say what exactly went on in your system. It wasn't clear what your conda environment looked like, and it (obviously) need to have python and pip installed. Presumably you already had this installed, but it wasn't entirely clear, and so I need to add this when creating the new environment.

If you didn't use the -DBUILD_SHARED_LIBS=ON you need to link to the static library (which you can do by also specifying --static to the python setup.py install). Additionally, when I did not use --use-pkg-config on the install option, it tried to compile still from the vendored igraph C core (from vendor/source/igraph). It's not exactly clear to me what went wrong.

All in all, the whole point is that the entire environment needs to be consistent: all the libraries and dependencies should be within the conda environment, otherwise you might run into some nasty problems.

@szhorvat I had to change the sugiyama layout algorithm which is written in the core C of igraph

Would this perhaps be something that could be more broadly useful for other people? If so, you could perhaps consider making a PR.

[AlexisPister]

I don't think I ever explicitly installed pip with conda on my previous environnements, but I could still use the command.

For the Sugiyama algorithm, I'll first see if it actually improves the layout and if yes I'll definitely consider making a PR.

[szhorvat]

For the Sugiyama algorithm, I'll first see if it actually improves the layout and if yes I'll definitely consider making a PR.

Please do 👍