rtree version 0.9 install error: OSError: libspatialindex_c.so: cannot open shared object file

[lior-k]

Hi, running pip install "rtree>=0.8,<1" fails with this error:

Collecting rtree<1,>=0.8
  Downloading https://files.pythonhosted.org/packages/5e/64/a01a6675c39ddfba2467cc6b432ce9af2e71a3a67e9e8ace106ccda10df8/Rtree-0.9.0.tar.gz (52kB)
    100% |████████████████████████████████| 61kB 2.7MB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-r2lpov2a/rtree/setup.py", line 3, in <module>
        import rtree
      File "/tmp/pip-build-r2lpov2a/rtree/rtree/__init__.py", line 1, in <module>
        from .index import Rtree
      File "/tmp/pip-build-r2lpov2a/rtree/rtree/index.py", line 6, in <module>
        from . import core
      File "/tmp/pip-build-r2lpov2a/rtree/rtree/core.py", line 104, in <module>
        rt = ctypes.CDLL(lib_name)
      File "/usr/local/lib/python3.6/ctypes/__init__.py", line 348, in __init__
        self._handle = _dlopen(self._name, mode)
    OSError: libspatialindex_c.so: cannot open shared object file: No such file or directory

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-r2lpov2a/rtree/

I installed libspatialindex_c.so manually by running the following, but the above error still happens even though /usr/local/lib/libspatialindex_c.so exists now :

LIBSPATIALINDEX_VERSION=spatialindex-src-1.8.5
curl -sLO "http://download.osgeo.org/libspatialindex/$LIBSPATIALINDEX_VERSION.tar.gz" && \
    tar xzf "$LIBSPATIALINDEX_VERSION.tar.gz" && \
    rm -rf "$LIBSPATIALINDEX_VERSION.tar.gz" && \
    cd "$LIBSPATIALINDEX_VERSION" && \
    ./configure && \
    make -j $( getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1 ) && \
    make install

[lior-k]

worth mentioning that pip install "rtree>=0.8,<0.9"works. this only happens on version 0.9

[DanielFEvans]

Seeing the same issue as of this morning on our CI server.

The system (running Scientific Linux 7) has libspatialindex v1.8.5 installed:

$ yum list installed | grep "spatialindex"
spatialindex.x86_64                   1.8.5-1.el7                 @epel

However, I have no shared library named libspatialindex_c.so, but instead libspatialindex_c.so.4:

$ locate spatialindex_c
/usr/lib64/libspatialindex_c.so.4
/usr/lib64/libspatialindex_c.so.4.1.0

The changes in this diff resulted in rtree being hardcoded to import spatialindex_c.so, rather than calling ctypes.util.find_library("spatialindex_c"), which was previously returning the correct name:

>>> import ctypes.util
>>> ctypes.util.find_library("spatialindex_c")
'libspatialindex_c.so.4'

[hobu]

Sorry for the churn. I was trying to clean up a messy area of the code that seemed to be getting patched by downstream packaging groups.

https://github.com/Toblerity/rtree/issues/56 implies that find_library doesn't find things in non-standard locations. Is that still true?

[hobu]

@DanielFEvans can you please try current master? If it fixes your issue, I will cut a 0.9.1 release immediately.

[DanielFEvans]

Appears that there was a change in Python 3.6 to cause find_library to search LD_LIBRARY_PATH: https://hg.python.org/cpython/rev/385181e809bc

I wonder if you could make more people happy with a fallback if find_library doesn't get a result:

lib_name = find_library('spatialindex_c')
if not lib_name:
    if 'linux' in sys.platform:
        lib_name = 'libspatialindex_c.so'
    elif 'darwin' in sys.platform:
        lib_name = 'libspatialindex_c.dylib'
    else:
        lib_name = 'libspatialindex_c'

Of course, that still doesn't work if you've got a libspatialindex_c.so.4 in a non-standard location (luckily, I don't!).

[hobu]

suggestion added in b85e2c8c426d3427b99af76ef3928b4644677adb

[DanielFEvans]

@hobu - latest master installs without error, unlike 0.9.0. Thanks!

[hobu]

0.9.1 released to fix this issue. https://pypi.org/project/Rtree/

[davidedelvento]

Mmmm, this is still failing for me. It used to work with rtree v0.8.3 but now it fails with v0.9.1 (I am doing exactly the same thing, just with a different version of rtree since I install on servers with ansible)

[hobu]

0.9.2 was just released. Is that what is failing for you? If so, I need more information. The 0.9.2 code is an adaptation of what was there in 0.8.3.

[davidedelvento]

Thanks for the quick response. Yes, it is still failing with 0.9.2

My ansible script fails as follow:

pip3 install numpy scipy folium fiona shapely cesiumpy matplotlib pandas netCDF4 rtree geopandas plotly sklearn keras tensorflow pysal

and it fails with

fatal: [IP-NUMBER-EDITED]: FAILED! => {
    "changed": false, 
    "cmd": "/usr/local/bin/pip3 install numpy scipy folium fiona shapely cesiumpy matplotlib pandas netCDF4 rtree geopandas pl
otly sklearn keras tensorflow pysal"
}

MSG:

stdout: Collecting numpy
  Using cached https://files.pythonhosted.org/packages/ab/e9/2561dbfbc05146bffa02167e09b9902e273decb2dc4cd5c43314ede20312/nump
y-1.17.4-cp35-cp35m-manylinux1_x86_64.whl
Collecting scipy
  Using cached https://files.pythonhosted.org/packages/1e/9c/4d7a573d49c449c95d33df6648a62a522ec46403f13ffe2c8cd3f094f746/scip
y-1.3.3-cp35-cp35m-manylinux1_x86_64.whl
Collecting folium
  Using cached https://files.pythonhosted.org/packages/fd/a0/ccb3094026649cda4acd55bf2c3822bb8c277eb11446d13d384e5be35257/foli
um-0.10.1-py2.py3-none-any.whl
Collecting fiona
  Using cached https://files.pythonhosted.org/packages/03/68/d6a70cd10afc31c336d69e13c371ff0088e14390b310da1668a648afb19d/Fion
a-1.8.13-cp35-cp35m-manylinux1_x86_64.whl
Collecting shapely
  Using cached https://files.pythonhosted.org/packages/a2/6c/966fa320a88fc685c956af08135855fa84a1589631256abebf73721c26ed/Shap
ely-1.6.4.post2-cp35-cp35m-manylinux1_x86_64.whl
Collecting cesiumpy
  Using cached https://files.pythonhosted.org/packages/20/4d/079f09eed2ab0123c21a4728470f3c62b6bf5f6c7bf00d879f6dc41bcd3d/cesi
umpy-0.3.3.tar.gz
Collecting matplotlib
  Using cached https://files.pythonhosted.org/packages/89/61/465fb3bfba684b0f53b5c4829c3c89e86e6fe9fdcdfda93e38f1788090f0/matp
lotlib-3.0.3-cp35-cp35m-manylinux1_x86_64.whl
Collecting pandas
  Using cached https://files.pythonhosted.org/packages/74/24/0cdbf8907e1e3bc5a8da03345c23cbed7044330bb8f73bb12e711a640a00/pand
as-0.24.2-cp35-cp35m-manylinux1_x86_64.whl
Collecting netCDF4
  Using cached https://files.pythonhosted.org/packages/4c/d3/b42ffc1a0b89df26b9e7796e5b79155e7d89e8b50565699c06ce8d5907df/netC
DF4-1.5.3-cp35-cp35m-manylinux1_x86_64.whl
Collecting rtree
  Using cached https://files.pythonhosted.org/packages/6b/11/78e7cec7e502c2db845b931c4f26863dc6bb493c022883affea161ad6363/Rtre
e-0.9.2.tar.gz

:stderr:     ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3064k240/rtree/setup.
py'"'"'; __file__='"'"'/tmp/pip-install-3064k240/rtree/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=
f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /t
mp/pip-install-3064k240/rtree/pip-egg-info
         cwd: /tmp/pip-install-3064k240/rtree/
    Complete output (13 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-3064k240/rtree/setup.py", line 3, in <module>
        import rtree
      File "/tmp/pip-install-3064k240/rtree/rtree/__init__.py", line 1, in <module>
        from .index import Rtree
      File "/tmp/pip-install-3064k240/rtree/rtree/index.py", line 6, in <module>
        from . import core
      File "/tmp/pip-install-3064k240/rtree/rtree/core.py", line 139, in <module>
        rt = ctypes.CDLL(lib_name)
      File "/usr/lib/python3.5/ctypes/__init__.py", line 347, in __init__
        self._handle = _dlopen(self._name, mode)
    OSError: libspatialindex_c.so.4: cannot open shared object file: No such file or directory
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Attempting the same thing manually yields:

$ pip3 install rtree
Collecting rtree
  Downloading https://files.pythonhosted.org/packages/6b/11/78e7cec7e502c2db845b931c4f26863dc6bb493c022883affea161ad6363/Rtree
-0.9.2.tar.gz (515kB)
     |████████████████████████████████| 522kB 3.7MB/s
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-q6lf6jup/rtree/setup.
py'"'"'; __file__='"'"'/tmp/pip-install-q6lf6jup/rtree/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=
f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /t
mp/pip-install-q6lf6jup/rtree/pip-egg-info
         cwd: /tmp/pip-install-q6lf6jup/rtree/
    Complete output (13 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-q6lf6jup/rtree/setup.py", line 3, in <module>
        import rtree
      File "/tmp/pip-install-q6lf6jup/rtree/rtree/__init__.py", line 1, in <module>
        from .index import Rtree
      File "/tmp/pip-install-q6lf6jup/rtree/rtree/index.py", line 6, in <module>
        from . import core
      File "/tmp/pip-install-q6lf6jup/rtree/rtree/core.py", line 139, in <module>
        rt = ctypes.CDLL(lib_name)
      File "/usr/lib/python3.5/ctypes/__init__.py", line 347, in __init__
        self._handle = _dlopen(self._name, mode)
    OSError: libspatialindex_c.so.4: cannot open shared object file: No such file or directory
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

but note that the lib is there:

$ ls /usr/local/lib/libspatialindex* -l
-rw-r--r-- 1 root root 1702694 Dec  9 17:45 /usr/local/lib/libspatialindex.a
-rw-r--r-- 1 root root  414536 Dec  9 17:45 /usr/local/lib/libspatialindex_c.a
-rwxr-xr-x 1 root root    1064 Dec  9 17:45 /usr/local/lib/libspatialindex_c.la
lrwxrwxrwx 1 root root      26 Dec  9 17:45 /usr/local/lib/libspatialindex_c.so -> libspatialindex_c.so.4.0.1
lrwxrwxrwx 1 root root      26 Dec  9 17:45 /usr/local/lib/libspatialindex_c.so.4 -> libspatialindex_c.so.4.0.1
-rwxr-xr-x 1 root root  218944 Dec  9 17:45 /usr/local/lib/libspatialindex_c.so.4.0.1
-rwxr-xr-x 1 root root    1016 Dec  9 17:45 /usr/local/lib/libspatialindex.la
lrwxrwxrwx 1 root root      24 Dec  9 17:45 /usr/local/lib/libspatialindex.so -> libspatialindex.so.4.0.1
lrwxrwxrwx 1 root root      24 Dec  9 17:45 /usr/local/lib/libspatialindex.so.4 -> libspatialindex.so.4.0.1
-rwxr-xr-x 1 root root  918504 Dec  9 17:45 /usr/local/lib/libspatialindex.so.4.0.1

Thanks again and let me know if you need anything more than this.

[hobu]

@davidedelvento which platform and packages do you have installed? I just added ubuntu 16.04 and 18.04 pip tests that install and run correctly.

[hobu]

See https://github.com/Toblerity/rtree/pull/131/files for the libspatialindex package I am installing on ubuntu.

[hobu]

The package that must be installed on trusty is libspatialindex-c3 and libspatialindex-c4v5 on 16.04 and 18.04

[davidedelvento]

@hobu thanks a lot for this quick response. I am running on Ubuntu 16.04.6 LTS and installing libspatialindex v1.8.5 from sources (I do not recall why at the moment and the commit comment on the ansible script that does that install does not clarify why not using the distro for this package). Right now I have troubles with testing that because of VMs start-up time, but I will get to it after a few meetings.

Meanwhile, though, I do not understand how install from source should cause this issue (which was not happening in rtree v0.8.3). As you can see with the logs above, the libraries are there and the error says No such file or directory. Do you confirm that rtree v0.9.2 is still looking into /usr/local/lib/ for libspatialindex?

[hobu]

If you're building from source, use the latest 1.9.3 if you can. https://libspatialindex.org/

I don't understand how rtree 0.8.x specifically looked in /usr/local/lib. It does have a SPATIALINDEX_C_LIBRARY environment variable you can set to provide a hint (which was added back in 0.9.2). If it did specific looking in /usr/local/lib, think it is bad taste for us to be doing that. I think you should have to modify your environment with LD_LIBRARY_PATH and ldconfig so that the Python find_library function can find your libspatialindex_c.so.

[hobu]

I think I have it figured out now in #131 by building a test case. Release coming today.