search

conda-forge / rasterio-feedstock

A conda-smithy repository for rasterio.
BSD 3-Clause "New" or "Revised" License
43 stars 23 forks source link

libabsl_cord.so.2301.0.0: cannot open shared object file #274

Closed rphel closed 1 year ago

rphel commented 1 year ago

Solution to issue cannot be found in the documentation.

Issue

When building our software we encouter this error with rasterio (1.3.8-py310hd227816_0) linux libraries:

 import rasterio
build.py:228
../../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/__init__.py:28: in <module>                                                                           
               from rasterio._version import gdal_version, get_geos_version, get_proj_version                                                                    
ImportError: libabsl_cord.so.2301.0.0: cannot open shared object file: No such file or directory

Both rasterio and GDAL libraries seem affected:

./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/_base.cpython-310-x86_64-linux-gnu.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/_env.cpython-310-x86_64-linux-gnu.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/_err.cpython-310-x86_64-linux-gnu.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/_features.cpython-310-x86_64-linux-gnu.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/_filepath.cpython-310-x86_64-linux-gnu.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/_fill.cpython-310-x86_64-linux-gnu.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/_io.cpython-310-x86_64-linux-gnu.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/_transform.cpython-310-x86_64-linux-gnu.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/_version.cpython-310-x86_64-linux-gnu.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/_warp.cpython-310-x86_64-linux-gnu.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/crs.cpython-310-x86_64-linux-gnu.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/rasterio/shutil.cpython-310-x86_64-linux-gnu.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libtiledb.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libtiledb.so.2.16
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libgdal.so
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libgdal.so.33
    libabsl_cord.so.2301.0.0 => not found
./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libgdal.so.33.3.7.1

The version that is installed is:

libabsl_cord.so.2308.0.0

Installed packages

#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge  
_openmp_mutex             4.5                       2_gnu  
affine                    2.4.0              pyhd8ed1ab_0  
attrs                     23.1.0             pyh71513ae_1  
blosc                     1.21.5               h0f2a231_0  
boost-cpp                 1.78.0               h2c5509c_4  
bzip2                     1.0.8                h7f98852_4  
c-ares                    1.19.1               hd590300_0  
ca-certificates           2023.7.22            hbcca054_0  
cairo                     1.16.0            h0c91306_1017  
certifi                   2023.7.22          pyhd8ed1ab_0  
cfitsio                   4.3.0                hbdc6101_0  
click                     8.1.7           unix_pyh707e725_0  
click-plugins             1.1.1                      py_0  
cligj                     0.7.2              pyhd8ed1ab_1  
colorama                  0.4.6              pyhd8ed1ab_0  
exceptiongroup            1.1.3              pyhd8ed1ab_0  
expat                     2.5.0                hcb278e6_1  
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0  
font-ttf-inconsolata      3.000                h77eed37_0  
font-ttf-source-code-pro  2.038                h77eed37_0  
font-ttf-ubuntu           0.83                 hab24e00_0  
fontconfig                2.14.2               h14ed4e7_0  
fonts-conda-ecosystem     1                             0  
fonts-conda-forge         1                             0  
freetype                  2.12.1               hca18f0e_1  
freexl                    1.0.6                h166bdaf_1  
geos                      3.12.0               h59595ed_0  
geotiff                   1.7.1               h22adcc9_11  
gettext                   0.21.1               h27087fc_0  
giflib                    5.2.1                h0b41bf4_3  
gtest                     1.14.0               h00ab1b0_1  
hdf4                      4.2.15               h501b40f_6  
hdf5                      1.14.2          nompi_h4f84152_100  
icu                       73.2                 h59595ed_0  
iniconfig                 2.0.0              pyhd8ed1ab_0  
json-c                    0.17                 h7ab15ed_0  
kealib                    1.5.1                hcd42e92_5  
keyutils                  1.6.1                h166bdaf_0  
krb5                      1.21.2               h659d440_0  
lcms2                     2.15                 haa2dc70_1  
ld_impl_linux-64          2.40                 h41732ed_0  
lerc                      4.0.0                h27087fc_0  
libabseil                 20230802.0      cxx17_h59595ed_3  
libaec                    1.0.6                hcb278e6_1  
libarchive                3.6.2                h039dbb9_1  
libblas                   3.9.0           18_linux64_openblas  
libcblas                  3.9.0           18_linux64_openblas  
libcrc32c                 1.1.2                h9c3ff4c_0  
libcurl                   8.2.1                hca28451_0  
libdeflate                1.18                 h0b41bf4_0  
libedit                   3.1.20191231         he28a2e2_2  
libev                     4.33                 h516909a_1  
libexpat                  2.5.0                hcb278e6_1  
libffi                    3.4.2                h7f98852_5  
libgcc-ng                 13.2.0               h807b86a_0  
libgdal                   3.7.1                h880a63b_9  
libgfortran-ng            13.2.0               h69a702a_0  
libgfortran5              13.2.0               ha4646dd_0  
libglib                   2.78.0               hebfc3b9_0  
libgomp                   13.2.0               h807b86a_0  
libgoogle-cloud           2.12.0               h8d7e28b_2  
libgrpc                   1.57.0               ha4d0f93_1  
libiconv                  1.17                 h166bdaf_0  
libjpeg-turbo             2.1.5.1              h0b41bf4_0  
libkml                    1.3.0             h37653c0_1015  
liblapack                 3.9.0           18_linux64_openblas  
libnetcdf                 4.9.2           nompi_h80fb2b6_112  
libnghttp2                1.52.0               h61bc06f_0  
libnsl                    2.0.0                h7f98852_0  
libopenblas               0.3.24          pthreads_h413a1c8_0  
libpng                    1.6.39               h753d276_0  
libpq                     15.4                 hfc447b1_0  
libprotobuf               4.23.4               hf27288f_5  
librttopo                 1.1.0               hb58d41b_14  
libspatialite             5.0.1               h15f6e67_28  
libsqlite                 3.43.0               h2797004_0  
libssh2                   1.11.0               h0841786_0  
libstdcxx-ng              13.2.0               h7e041cc_0  
libtiff                   4.5.1                h8b53f26_1  
libuuid                   2.38.1               h0b41bf4_0  
libwebp-base              1.3.1                hd590300_0  
libxcb                    1.15                 h0b41bf4_0  
libxml2                   2.11.5               h232c23b_1  
libzip                    1.10.1               h2629f0a_3  
libzlib                   1.2.13               hd590300_5  
lz4-c                     1.9.4                hcb278e6_0  
lzo                       2.10              h516909a_1000  
markdown-it-py            3.0.0              pyhd8ed1ab_0  
mdurl                     0.1.0              pyhd8ed1ab_0  
ncurses                   6.4                  hcb278e6_0  
nspr                      4.35                 h27087fc_0  
nss                       3.92                 h1d7d5a4_0  
numpy                     1.25.2          py310ha4c1d20_0  
openjpeg                  2.5.0                hfec8fc6_2  
openssl                   3.1.2                hd590300_0  
packaging                 23.1               pyhd8ed1ab_0  
pcre2                     10.40                hc3806b6_0  
pip                       23.2.1             pyhd8ed1ab_0  
pixman                    0.40.0               h36c2ea0_0  
pluggy                    1.3.0              pyhd8ed1ab_0  
poppler                   23.08.0              hd18248d_0  
poppler-data              0.4.12               hd8ed1ab_0  
postgresql                15.4                 h8972f4a_0  
proj                      9.2.1                ha643af7_0  
pthread-stubs             0.4               h36c2ea0_1001  
pygments                  2.16.1             pyhd8ed1ab_0  
pyparsing                 3.1.1              pyhd8ed1ab_0  
pytest                    7.4.2              pyhd8ed1ab_0  
python                    3.10.12         hd12c33a_0_cpython  
python_abi                3.10                    3_cp310  
pyyaml                    6.0.1           py310h2372a71_0  
rasterio                  1.3.8           py310hd227816_0  
re2                       2023.03.02           h8c504da_0  
readline                  8.2                  h8228510_1  
rich                      13.5.1             pyhd8ed1ab_0  
setuptools                68.1.2             pyhd8ed1ab_0  
snappy                    1.1.10               h9fff704_0  
snuggs                    1.4.7                      py_0  
sqlite                    3.43.0               h2c6b66d_0  
tiledb                    2.16.3               h84d19f0_1  
tk                        8.6.12               h27826a3_0  
tomli                     2.0.1              pyhd8ed1ab_0  
typing_extensions         4.7.1              pyha770c72_0  
tzcode                    2023c                h0b41bf4_0  
tzdata                    2023c                h71feb2d_0  
wheel                     0.41.2             pyhd8ed1ab_0  
xerces-c                  3.2.4                hac6953d_3  
xorg-kbproto              1.0.7             h7f98852_1002  
xorg-libice               1.1.1                hd590300_0  
xorg-libsm                1.2.4                h7391055_0  
xorg-libx11               1.8.6                h8ee46fc_0  
xorg-libxau               1.0.11               hd590300_0  
xorg-libxdmcp             1.1.3                h7f98852_0  
xorg-libxext              1.3.4                h0b41bf4_2  
xorg-libxrender           0.9.11               hd590300_0  
xorg-renderproto          0.11.1            h7f98852_1002  
xorg-xextproto            7.3.0             h0b41bf4_1003  
xorg-xproto               7.0.31            h7f98852_1007  
xz                        5.2.6                h166bdaf_0  
yaml                      0.2.5                h7f98852_2  
zlib                      1.2.13               hd590300_5  
zstd                      1.5.5                hfc55251_0

Environment info

active environment : ./conda-bld/mysoftware-split_1694441069965/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac
    active env location : ./conda-bld/mysoftware-split_1694441069965/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac
            shell level : 3
       user config file : /home/myusername/.condarc
 populated config files : /home/myusername/.condarc
          conda version : 23.7.3
    conda-build version : 3.26.1
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.0=0
                          __glibc=2.31=0
                          __linux=5.15.0=0
                          __unix=0=0
       base environment : /data/myusername/mambaforge  (writable)
      conda av data dir : /data/myusername/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://myartifactory/artifactory/api/conda/conda-forge/linux-64
                          https://myartifactory/artifactory/api/conda/conda-forge/noarch
          package cache : /data/myusername/mambaforge/pkgs
                          /home/myusername/.conda/pkgs
       envs directories : /data/myusername/mambaforge/envs
                          /home/myusername/.conda/envs
               platform : linux-64
             user-agent : conda/23.7.3 requests/2.31.0 CPython/3.10.12 Linux/5.15.0-78-generic ubuntu/20.04.6 glibc/2.31
                UID:GID : 505:102
           offline mode : False
ocefpaf commented 1 year ago

The channel column on your package list doesn't show conda-forge. I wonder if those packages are coming from defaults.

rphel commented 1 year ago

It is not possible as my computer is in a private network, I'm only using a mirror to access conda-forge repository (https://myartifactory/artifactory/api/conda/conda-forge/linux-64).

rphel commented 1 year ago

Surprisingly, rasterio doesn't depend on libaseil officialy even if libraries are linked against it. Only protobuf is listed by conda-tree:

 conda-tree -p ./_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/ whoneeds libabseil
libprotobuf
libgrpc
libgoogle-cloud
ocefpaf commented 1 year ago

It is not possible as my computer is in a private network, I'm only using a mirror to access conda-forge repository (https://myartifactory/artifactory/api/conda/conda-forge/linux-64).

We don't know the status of that mirror and if it propagates things like repodatapatches. I believe you are in an unsupported side of things. With that said, I can reproduce that with our own packages and I'm not sure what may have caused it. Investigating...

akrherz commented 1 year ago

My GHA CI just hit this as well :/ An python 3.10 environmental diff between a job that worked 3 hours ago

$ diff /tmp/good /tmp/bad
67a68
> gtest                     1.14.0               h00ab1b0_1    conda-forge
85c86
< libabseil                 20230125.3      cxx17_h59595ed_0    conda-forge
---
> libabseil                 20230802.0      cxx17_h59595ed_3    conda-forge
88c89
< libarrow                  13.0.0           h1ed0495_3_cpu    conda-forge
---
> libarrow                  13.0.0           h1935d02_4_cpu    conda-forge
108,109c109,110
< libgoogle-cloud           2.12.0               h840a212_1    conda-forge
< libgrpc                   1.56.2               h3905398_1    conda-forge
---
> libgoogle-cloud           2.12.0               h8d7e28b_2    conda-forge
> libgrpc                   1.57.0               ha4d0f93_1    conda-forge
121c122
< libprotobuf               4.23.3               hd1fb520_1    conda-forge
---
> libprotobuf               4.23.4               hf27288f_5    conda-forge
155c156
< orc                       1.9.0                h385abfd_1    conda-forge
---
> orc                       1.9.0                h52d3b3c_2    conda-forge
173c174
< pyarrow                   13.0.0          py310hf9e7431_3_cpu    conda-forge
---
> pyarrow                   13.0.0          py310hf9e7431_4_cpu    conda-forge
ocefpaf commented 1 year ago

@hmaarrfk sorry for the random ping but maybe you know what may have changed in the underlying dependencies here?

akrherz commented 1 year ago

downgrading to libgrpc=1.56.2 results in a working minimal conda-forge environment for rasterio.

ocefpaf commented 1 year ago

downgrading to libgrpc=1.56.2 results in a working minimal conda-forge environment for rasterio.

That is why I asked Mark above. I wonder if we missed some pinning somewhere...

akrherz commented 1 year ago

The dep path appears to be rasterio -> libgdal -> tiledb -> libgoogle-cloud -> libgrpc , perhaps we need this specified within gdal?

hmaarrfk commented 1 year ago

I'm hitting this reliably on https://github.com/conda-forge/isce2-feedstock/pull/65 sorry i've been debugging it too. no clue as of yet.

hmaarrfk commented 1 year ago

unfortunately, this will have to wait for @h-vetinari's input....

akrherz commented 1 year ago

@hmaarrfk , perhaps this will be fixed by conda-forge/tiledb-feedstock#209

h-vetinari commented 1 year ago

I already replied in a bunch of places for this, but yes, https://github.com/conda-forge/tiledb-feedstock/pull/209 should fix this for the most part. Someone still has to go write a repodata patch for the old tiledb builds that don't have an libabseil-dependence, and will (probably...) continue to be wrongly considered by the solver for newer abseil, unless we patch in a run-dep for libabseil 20230125.

akrherz commented 1 year ago

@rphel , can you verify that this issue has gone away now with tiledb builds 2 and 3 of version 2.16.3 now available? I am unable to locally reproduce anymore.

rphel commented 1 year ago

Yes it works thanks!