ImportError: libgip.so: cannot open shared object file: No such file or directory

[benhosmer]

This happens on master as well as develop:

#!/bin/bash

$PYTHON setup.py build
$PYTHON setup.py install

ls lib/python2.7/site-packages/gippy
algorithms.py   _algorithms.so  gippy.pyc  __init__.py   libgip.so  test.pyc    version.pyc
algorithms.pyc  gippy.py        _gippy.so  __init__.pyc  test.py    version.py

Python 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:09:15) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import numpy
>>> import potrace
>>> import gippy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vagrant/miniconda2/envs/bfalg/lib/python2.7/site-packages/gippy/__init__.py", line 41, in <module>
    from .gippy import init, DataType, GeoImage, GeoVector, Options
  File "/home/vagrant/miniconda2/envs/bfalg/lib/python2.7/site-packages/gippy/gippy.py", line 30, in <module>
    _gippy = swig_import_helper()
  File "/home/vagrant/miniconda2/envs/bfalg/lib/python2.7/site-packages/gippy/gippy.py", line 26, in swig_import_helper
    _mod = imp.load_module('_gippy', fp, pathname, description)
ImportError: libgip.so: cannot open shared object file: No such file or directory

[ircwaves]

I've seen that happen when you run from the cloned repo directory. Is that where you ran the import gippy statement above?

[matthewhanson]

@benhosmer as @ircwaves said if you run it from that directory when it imports it may be trying to import from the wrong location.

Otherwise, it could be an issue with the specific platform, and I can test on a Red Hat image.

I've been working on a better way to load the libgip library, but for now another possible workaround is to make libgip.so available as it it were a system package, so copying it into /usr/lib or adding it's location to the LD_LIBRARY_PATH.

[benhosmer]

@matthewhanson unfortunately in my case, I can't install to /usr/lib. I don't have write to the system. We also can't use LD_LIBRARY_PATH.

This is using miniconda instead of pip, but I suspect the issue would happen in a virtualenv using pip as well.

I've seen that happen when you run from the cloned repo directory. Is that where you ran the import gippy statement above?

No, this happens regardless of where I invoke the python interpreter.

[matthewhanson]

@benhosmer this one had me chasing my tail for a while. The loops the gippy setup has to go through in order to properly set the library path for libgip.so from _gippy.so is complicated due to limitations in python-setuptools. I've been looking for an alternative using ctypes and dynamically loading libgip.so in init.py but have never been able to get it to work. I was looking at that again and reviewing the setup file, when I realized the problem.

GIPPY needs to set the runtime library directory in the _gippy.so and _algorithms.so binary libraries with the location of libgip.so. However, the location isn't actually known until install time, so I overrode the python-setuptools classes to call the build step (build_ext) from the install run function, after setting a global variable containing the install location. However, none of this worked for mac, which required setting a special variable called @loader_path as the rpath in the original binary. @loader_path looks for dependent libs in the same directory as the lib itself....very handy! Then, at some later date I found out about $ORIGIN, which is like @loader_path, but for Linux. By setting that in the library I avoided having to hard code the path.

The bottom line here is that the reason it fails, is because you are calling: python setup.py build

then separately calling python setup.py install

When you call install it has already built the extension, so the hacky workflow fails, and the rpath in _gippy.so is not set. For now, you can fix this just by, with a clean repo, install python setup.py install

Or, just install via pip from the repo directory: pip install .

Can also install straight from github: pip install git+https://github.com/gipit/gippy.git@develop

I'm currently trying to simplify things though, since I made the update to use $ORIGIN, I'm not sure the extra complexity and calling build_ext from the install setup class is needed, I'm exploring further.

[benhosmer]

As an aside, I can't use pip, which is the reason for using conda.

I changed the build to just do python setup.py install, but now I get:

>>> import gippy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vagrant/miniconda2/envs/bfalg2/lib/python2.7/site-packages/gippy/__init__.py", line 41, in <module>
    from .gippy import init, DataType, GeoImage, GeoVector, Options
  File "/home/vagrant/miniconda2/envs/bfalg2/lib/python2.7/site-packages/gippy/gippy.py", line 30, in <module>
    _gippy = swig_import_helper()
  File "/home/vagrant/miniconda2/envs/bfalg2/lib/python2.7/site-packages/gippy/gippy.py", line 26, in swig_import_helper
    _mod = imp.load_module('_gippy', fp, pathname, description)
ImportError: libgdal.so.1: cannot open shared object file: No such file or directory

I don't see GDAL declared in any requirements for gippy or pypotrace, but it looks like gippy does require gdal. What verson? I'm using 2.1.3 right now because it is available in anaconda, but I don't see a libgdal.so.1.

[matthewhanson]

@benhosmer GDAL is a system dependency, so not declared in a requirements.txt file which is for python dependencies. The GIPPY documentation has the system dependencies required. https://gippy.readthedocs.io/en/latest/install.html

It will dynamically link against whatever version is was built against....so if it built properly (check the output from the install step) it should have the proper GDAL version. Does libgdal not exist at all anywhere? It will work with either GDAL v1 or v2.

[matthewhanson]

@benhosmer I also recall that, if GDAL was compiled and installed rather than a pre-built package, it likely put it in /usr/local/lib which may not be in your path. I've seen this on some linux distros. If libgdal is there and you don't have a way to include it in your path, I could add it as a runtime library to the built gippy libraries. It wouldn't hurt to include it as a runtime path, since it's common gdal would be installed there and removes the need to use LD_LIBRARY_PATH

[benhosmer]

GDAL exists, and when I install from the built packages I have the following in lib/:

...
libgdal.a                    libiomp5.so                  libncursesw.so.5          libtinfow.so.5.9
libgdal.la                   libjbig85.a                  libncursesw.so.5.9        libtk8.5.so
libgdal.so                   libjbig.a                    libnetcdf.a               libtkstub8.5.a
libgdal.so.20                libjpeg.a                    libnetcdf.la              libxerces-c-3.1.so
libgdal.so.20.1.0            libjpeg.la                   libnetcdf.settings        libxerces-c.la
libgdal.so.20.1.3            libjpeg.so                   libnetcdf.so              libxerces-c.so
libgeos-3.5.0.so 
...

Changing my build to just python setup.py install didn't help:

>>> import potrace
>>> import numpy
>>> import gdal
>>> import gippy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vagrant/miniconda2/envs/bfalg2/lib/python2.7/site-packages/gippy/__init__.py", line 26, in <module>
    from .gippy import init, DataType, GeoImage, GeoVector, Options
  File "/home/vagrant/miniconda2/envs/bfalg2/lib/python2.7/site-packages/gippy/gippy.py", line 30, in <module>
    _gippy = swig_import_helper()
  File "/home/vagrant/miniconda2/envs/bfalg2/lib/python2.7/site-packages/gippy/gippy.py", line 26, in swig_import_helper
    _mod = imp.load_module('_gippy', fp, pathname, description)
ImportError: libgip.so: cannot open shared object file: No such file or directory

[ircwaves]

@benhosmer -- does your build env get completely wiped each time? (like git clean -xfd) Maybe you've got some stale objects?

[benhosmer]

@matthewhanson if it helps in your troubleshooting of this, I've got a README about how to duplicate my conda environment: https://github.com/venicegeo/pzsvc-ndwi-py/tree/pipeline-refactor/conda-recipes

As well as the recipes I'm using. Right now I have to build agg, pypotrace, and gippy from source, since those aren't in any of the anaconda channels.

[benhosmer]

@ircwaves I assume it does, because I run conda build clean and conda build purge. Also of note, pip is not an option because of the environment. I'm not using a git repo, but just a snapshot of the download.zip for develop.

[matthewhanson]

@benhosmer In two of your comments above, you first said you did setup install and got the the libgdal missing, and then in a later comment with setup install you got the libgip missing error. If you got to libgdal that means it did find libgip, so looks like something happened between those.

At any rate, I will check out the conda environment and try to replicate this myself to see if I come up with anything.

[benhosmer]

I might save you a little bit of work. I ran conda build with debug output on, and saw this:

Fixing permissions
Detected hard-coded path in binary file lib/python2.7/site-packages/gippy/_algorithms.so
Detected hard-coded path in binary file lib/python2.7/site-packages/gippy/_gippy.so
Detected hard-coded path in binary file lib/python2.7/site-packages/gippy/libgip.so

The solution might be a build flag that I found in the docs. I'm looking into has_prefix_files and relocation.

[benhosmer]

That did it. I had to specify binary_relocation: False in my conda meta.yaml file:

...
build:
  string: py{{py}}_0
  binary_relocation: False
...

[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import numpy
>>> import potrace
>>> import gippy

[matthewhanson]

I have also simplified setup and tested that both 'install' and 'develop' work for building, on osx and linux, and for python 2.7, 3.4, and 3.5 (python versions included in travis builds now, osx is still not). Also, if building the extension separately from installation the path is now properly set in the libraries, as identified in this issue.

PR: https://github.com/gipit/gippy/pull/124