import fiona fails with ImportError: libpoppler.so.71: cannot open shared object file: No such file or directory

[akrherz]

My fiona install from conda-forge was working up until a few days ago, but with a recent update it fails.

$ conda create -n TEST python=3.6 fiona
$ conda activate TEST
$ python -c 'import fiona'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/miniconda3/envs/TEST/lib/python3.6/site-packages/fiona/__init__.py", line 69, in <module>
    from fiona.collection import Collection, BytesCollection, vsi_path
  File "/opt/miniconda3/envs/TEST/lib/python3.6/site-packages/fiona/collection.py", line 9, in <module>
    from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
ImportError: libpoppler.so.71: cannot open shared object file: No such file or directory

[ocefpaf]

I cannot reproduce that on Linux (I guess you are on OS X, right?)

conda create -n TEST python=3.6 fiona
conda activate TEST
python -c 'import fiona; print(fiona.__version__)'
1.7.11

Here is my conda list:

> conda list
# packages in environment at /home/filipe/miniconda3/envs/TEST:
#
# Name                    Version                   Build  Channel
backcall                  0.1.0                      py_0    conda-forge
bleach                    2.1.3                      py_0    conda-forge
boost                     1.66.0                   py36_1    conda-forge
boost-cpp                 1.66.0                        1    conda-forge
bzip2                     1.0.6                         1    conda-forge
ca-certificates           2018.4.16                     0    conda-forge
cairo                     1.14.10                       0    conda-forge
certifi                   2018.4.16                py36_0    conda-forge
click                     6.7                        py_1    conda-forge
click-plugins             1.0.3                    py36_0    conda-forge
cligj                     0.4.0                    py36_0    conda-forge
curl                      7.60.0                        0    conda-forge
dbus                      1.11.0                        0    conda-forge
decorator                 4.3.0                      py_0    conda-forge
entrypoints               0.2.3                    py36_1    conda-forge
expat                     2.2.5                         0    conda-forge
fiona                     1.7.11                   py36_3    conda-forge
fontconfig                2.12.6                        0    conda-forge
freetype                  2.8.1                         0    conda-forge
freexl                    1.0.5                         0    conda-forge
gdal                      2.2.4                    py36_0    conda-forge
geos                      3.6.2                         1    conda-forge
geotiff                   1.4.2                         1    conda-forge
gettext                   0.19.8.1                      0    conda-forge
giflib                    5.1.4                         0    conda-forge
glib                      2.55.0                        0    conda-forge
gmp                       6.1.2                         0    conda-forge
gst-plugins-base          1.8.0                         0    conda-forge
gstreamer                 1.8.0                         1    conda-forge
hdf4                      4.2.13                        0    conda-forge
hdf5                      1.10.1                        2    conda-forge
html5lib                  1.0.1                      py_0    conda-forge
icu                       58.2                          0    conda-forge
ipykernel                 4.8.2                    py36_0    conda-forge
ipython                   6.4.0                    py36_0    conda-forge
ipython_genutils          0.2.0                    py36_0    conda-forge
ipywidgets                7.2.1                    py36_1    conda-forge
jedi                      0.12.0                   py36_0    conda-forge
jinja2                    2.10                     py36_0    conda-forge
jpeg                      9b                            2    conda-forge
json-c                    0.12.1                        0    conda-forge
jsonschema                2.6.0                    py36_1    conda-forge
jupyter                   1.0.0                      py_1    conda-forge
jupyter_client            5.2.3                    py36_0    conda-forge
jupyter_console           5.2.0                    py36_0    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
kealib                    1.4.7                         4    conda-forge
krb5                      1.14.6                        0    conda-forge
libdap4                   3.18.3                        2    conda-forge
libffi                    3.2.1                         3    conda-forge
libgcc                    7.2.0                h69d50b8_2    defaults
libgcc-ng                 7.2.0                hdf63c60_3    defaults
libgdal                   2.2.4                         2    conda-forge
libgfortran               3.0.0                         1    defaults
libgfortran-ng            7.2.0                hdf63c60_3    defaults
libiconv                  1.15                          0    conda-forge
libkml                    1.3.0                         6    conda-forge
libnetcdf                 4.6.1                         2    conda-forge
libopenblas               0.2.20               h9ac9557_4    defaults
libpng                    1.6.34                        0    conda-forge
libpq                     9.6.3                         0    conda-forge
libsodium                 1.0.16                        0    conda-forge
libspatialite             4.3.0a                       19    conda-forge
libssh2                   1.8.0                         2    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3    defaults
libtiff                   4.0.9                         0    conda-forge
libxcb                    1.13                          0    conda-forge
libxml2                   2.9.8                         0    conda-forge
markupsafe                1.0                      py36_0    conda-forge
mistune                   0.8.3                    py36_1    conda-forge
munch                     2.3.2                      py_0    conda-forge
nbconvert                 5.3.1                      py_1    conda-forge
nbformat                  4.4.0                    py36_0    conda-forge
ncurses                   5.9                          10    conda-forge
notebook                  5.5.0                    py36_0    conda-forge
numpy                     1.14.3           py36h28100ab_1    defaults
numpy-base                1.14.3           py36h0ea5e3f_1    defaults
openjpeg                  2.3.0                         2    conda-forge
openssl                   1.0.2o                        0    conda-forge
pandoc                    2.2.1                         0    conda-forge
pandocfilters             1.4.2                    py36_0    conda-forge
parso                     0.2.1                      py_0    conda-forge
pcre                      8.41                          1    conda-forge
pexpect                   4.5.0                    py36_0    conda-forge
pickleshare               0.7.4                    py36_0    conda-forge
pip                       9.0.3                    py36_0    conda-forge
pixman                    0.34.0                        2    conda-forge
poppler                   0.61.1                        3    conda-forge
poppler-data              0.4.9                         0    conda-forge
proj4                     4.9.3                         5    conda-forge
prompt_toolkit            1.0.15                   py36_0    conda-forge
ptyprocess                0.5.2                    py36_0    conda-forge
pygments                  2.2.0                    py36_0    conda-forge
pyqt                      5.6.0                    py36_5    conda-forge
python                    3.6.5                         1    conda-forge
python-dateutil           2.7.3                      py_0    conda-forge
pyzmq                     17.0.0                   py36_4    conda-forge
qt                        5.6.2                         7    conda-forge
qtconsole                 4.3.1                    py36_0    conda-forge
readline                  7.0                           0    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                39.2.0                   py36_0    conda-forge
shapely                   1.6.4                    py36_0    conda-forge
simplegeneric             0.8.1                    py36_0    conda-forge
sip                       4.18                     py36_1    conda-forge
six                       1.11.0                   py36_1    conda-forge
sqlite                    3.20.1                        2    conda-forge
terminado                 0.8.1                    py36_0    conda-forge
testpath                  0.3.1                    py36_0    conda-forge
tk                        8.6.7                         0    conda-forge
tornado                   5.0.2                    py36_0    conda-forge
traitlets                 4.3.2                    py36_0    conda-forge
util-linux                2.21                          0    defaults
wcwidth                   0.1.7                    py36_0    conda-forge
webencodings              0.5                      py36_0    conda-forge
wheel                     0.31.0                   py36_0    conda-forge
widgetsnbextension        3.2.1                    py36_0    conda-forge
xerces-c                  3.2.0                         0    conda-forge
xorg-libxau               1.0.8                         3    conda-forge
xorg-libxdmcp             1.1.2                         3    conda-forge
xz                        5.2.3                         0    conda-forge
zeromq                    4.2.5                         1    conda-forge
zlib                      1.2.11                        0    conda-forge

[akrherz]

Thanks @ocefpaf for the instant support again, you are wonderful. I am on RHEL 7.5 64bit, just did a clean all and tried again with the same failure.

$ conda create -n TEST python=3.6 fiona
Solving environment: done

## Package Plan ##

  environment location: /opt/miniconda3/envs/TEST

  added / updated specs: 
    - fiona
    - python=3.6

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    fontconfig-2.13.0          |       h9420a91_0         291 KB
    blas-1.0                   |              mkl           6 KB
    poppler-0.65.0             |       h581218d_1         1.6 MB
    freetype-2.9.1             |       h8a8886c_0         821 KB
    cairo-1.14.12              |       h8948797_3         1.3 MB
    numpy-1.14.3               |   py36hcd700cb_1          41 KB
    krb5-1.16                  |       h3f6afd0_6         1.3 MB
    libpq-10.3                 |       h1ad7b7a_0         2.7 MB
    ------------------------------------------------------------
                                           Total:         8.0 MB

The following NEW packages will be INSTALLED:

    blas:            1.0-mkl                
    bzip2:           1.0.6-h14c3975_5       
    ca-certificates: 2018.03.07-0           
    cairo:           1.14.12-h8948797_3     
    certifi:         2018.4.16-py36_0       
    click:           6.7-py36h5253387_0     
    click-plugins:   1.0.3-py36_0           
    cligj:           0.4.0-py36_0           
    curl:            7.60.0-h84994c4_0      
    expat:           2.2.5-he0dffb1_0       
    fiona:           1.7.10-py36h48a52f0_0  
    fontconfig:      2.13.0-h9420a91_0      
    freetype:        2.9.1-h8a8886c_0       
    freexl:          1.0.5-h14c3975_0       
    gdal:            2.2.2-py36hc209d97_1   
    geos:            3.6.2-heeff764_2       
    giflib:          5.1.4-h26a3ec6_1       
    glib:            2.56.1-h000015b_0      
    hdf4:            4.2.13-h3ca952b_2      
    hdf5:            1.10.1-h9caa474_1      
    icu:             58.2-h9c2bf20_1        
    intel-openmp:    2018.0.0-8             
    jpeg:            9b-h024ee3a_2          
    json-c:          0.12.1-ha6a3662_2      
    kealib:          1.4.7-h79811e5_5       
    krb5:            1.16-h3f6afd0_6        
    libboost:        1.65.1-habcd387_4      
    libcurl:         7.60.0-h1ad7b7a_0      
    libdap4:         3.19.0-h5bd89bb_2      
    libedit:         3.1.20170329-h6b74fdf_2
    libffi:          3.2.1-hd88cf55_4       
    libgcc-ng:       7.2.0-hdf63c60_3       
    libgdal:         2.2.2-h804cdde_1       
    libgfortran-ng:  7.2.0-hdf63c60_3       
    libkml:          1.3.0-h9d32c78_3       
    libnetcdf:       4.4.1.1-h816af47_8     
    libpng:          1.6.34-hb9fc6fc_0      
    libpq:           10.3-h1ad7b7a_0        
    libspatialite:   4.3.0a-h72746d6_18     
    libssh2:         1.8.0-h9cfc8f7_4       
    libstdcxx-ng:    7.2.0-hdf63c60_3       
    libtiff:         4.0.9-he85c1e1_1       
    libuuid:         1.0.3-h1bed415_2       
    libxcb:          1.13-h1bed415_1        
    libxml2:         2.9.8-h26e45fe_1       
    mkl:             2018.0.2-1             
    mkl_fft:         1.0.1-py36h3010b51_0   
    mkl_random:      1.0.1-py36h629b387_0   
    munch:           2.3.2-py36_0           
    ncurses:         6.1-hf484d3e_0         
    numpy:           1.14.3-py36hcd700cb_1  
    numpy-base:      1.14.3-py36h9be14a7_1  
    openjpeg:        2.3.0-h05c96fa_1       
    openssl:         1.0.2o-h20670df_0      
    pcre:            8.42-h439df22_0        
    pip:             10.0.1-py36_0          
    pixman:          0.34.0-hceecf20_3      
    poppler:         0.65.0-h581218d_1      
    poppler-data:    0.4.9-0                
    proj4:           4.9.3-hc8507d1_7       
    python:          3.6.5-hc3d631a_2       
    readline:        7.0-ha6073c6_4         
    setuptools:      39.1.0-py36_0          
    shapely:         1.6.4-py36h0c48222_0   
    six:             1.11.0-py36h372c433_1  
    sqlite:          3.23.1-he433501_0      
    tk:              8.6.7-hc745277_3       
    wheel:           0.31.1-py36_0          
    xerces-c:        3.2.1-hac72e42_0       
    xz:              5.2.4-h14c3975_4       
    zlib:            1.2.11-ha838bed_2  
Downloading and Extracting Packages
fontconfig-2.13.0    |  291 KB | ####################################### | 100% 
blas-1.0             |    6 KB | ####################################### | 100% 
poppler-0.65.0       |  1.6 MB | ####################################### | 100% 
freetype-2.9.1       |  821 KB | ####################################### | 100% 
cairo-1.14.12        |  1.3 MB | ####################################### | 100% 
numpy-1.14.3         |   41 KB | ####################################### | 100% 
krb5-1.16            |  1.3 MB | ####################################### | 100% 
libpq-10.3           |  2.7 MB | ####################################### | 100% 
Preparing transaction: done
Verifying transaction: - 
SafetyError: The package for proj4 located at /opt/miniconda3/pkgs/proj4-4.9.3-hc8507d1_7
appears to be corrupted. The path 'share/proj/epsg'
has a sha256 mismatch.
  reported sha256: 84ed2546df3994153a91e57d448650c0de4949c251badbb48f7a01a0385db4c9
  actual sha256: bcb99237f9652285b12a05cc9da643aa7d26179555d2151da15bc94e4a1d45bf

[sgillies]

@akrherz I had the same problem with the conda install I'm doing to build Rasterio docs on Read the Docs. In my case, the culprit was packages from the defaults channel, and I solved it in this commit: https://github.com/mapbox/rasterio/commit/55b38cf84a0d2141690583d2d58c168807d2ed98#diff-b7cb75ca59993a879e29bed8cd8469b8. My docs are building again.

[ocefpaf]

I do see many packages from defaults and that makes me think you have the conda-forge channel in a lower priority in your .condarc. (Same situation that @sgillies describes above.)

You sha256 is also related to a package from defaults, that may be a temporary network issue or a real problem with the package.

Please take a look at https://conda-forge.org/docs/conda-forge_gotchas.html#using-multiple-channels which describes the pitfalls of mixing channels and the best practices to avoid them.

[ceball]

@ocefpaf, I think you're right (i.e. I think this isn't an issue with the fiona-feedstock). However, I also think it's easy to mistakenly mix channels without realizing it even if you follow best practices (unless I'm missing something from the best practices doc).

I'm also on linux, and can reproduce ImportError: libpoppler.so.71: cannot open shared object file: No such file or directory if I install poppler without taking care about where gdal comes from.

$ cat ~/.condarc 
channels:
  - conda-forge
  - defaults

It's fine by default in reasonable isolation:

$ conda create -n testfionapoppler -c conda-forge python=3.6 numpy scipy fiona

$ conda activate testfionapoppler

(testfionapoppler)$ conda list fiona
# packages in environment at /cio/mc3/envs/testfionapoppler:
#
# Name                    Version                   Build  Channel
fiona                     1.7.11                   py36_3    conda-forge

(testfionapoppler)$ conda list poppler
# packages in environment at /cio/mc3/envs/testfionapoppler:
#
# Name                    Version                   Build  Channel
poppler                   0.61.1                        3    conda-forge
poppler-data              0.4.9                         0    conda-forge

(testfionapoppler)$ conda list gdal
# packages in environment at /cio/mc3/envs/testfionapoppler:
#
# Name                    Version                   Build  Channel
gdal                      2.2.4                    py36_0    conda-forge
libgdal                   2.2.4                         0    conda-forge

(testfionapoppler)$ python -c "import fiona; print(fiona.__version__)"
1.7.11

Simulate some action that causes gdal to switch to defaults:

(testfionapoppler)$ conda update -c conda-forge poppler numpy scipy

(testfionapoppler)$ conda list poppler
# packages in environment at /cio/mc3/envs/testfionapoppler:
#
# Name                    Version                   Build  Channel
poppler                   0.64.0                        0    conda-forge
poppler-data              0.4.9                         0    conda-forge

(testfionapoppler)$ conda list gdal
# packages in environment at /cio/mc3/envs/testfionapoppler:
#
# Name                    Version                   Build  Channel
gdal                      2.2.2            py36hc209d97_1
libgdal                   2.2.2                h804cdde_1

(testfionapoppler)$ python -c "import fiona; print(fiona.__version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/cio/mc3/envs/testfionapoppler/lib/python3.6/site-packages/fiona/__init__.py", line 69, in <module>
    from fiona.collection import Collection, BytesCollection, vsi_path
  File "/cio/mc3/envs/testfionapoppler/lib/python3.6/site-packages/fiona/collection.py", line 9, in <module>
    from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
ImportError: libpoppler.so.71: cannot open shared object file: No such file or directory

If I instead run conda update -c conda-forge gdal poppler numpy scipy, there's no problem (gdal isn't switched to defaults).

I could have avoided the problem by doing something like conda install conda-forge::gdal conda-forge::numpy conda-forge::scipy conda-forge::fiona in the first place. Then subsequent actions such as conda update poppler would not cause gdal to switch to defaults.

(Note: I've included numpy and scipy above because I was simultaneously debugging a different - though related - problem.)

[akrherz]

@ocefpaf thank you again, I switched the channel ordering and conda update --all got things working again for fiona. I really wish there was something that would save users like me, from themselves in this situation. If conda-forge is required to be higher priority than defaults, then perhaps the tooling should enforce that situation? I dunno

[ocefpaf]

This is an issue that we are battling for a long time now and conda's solver does not help us much, sometimes even the channel priority you used above fails :unamused:

The situation should improved once conda-forge moves to conda-build 3.

Closing this but feel free to open if you experience issues again.

[ocefpaf]

If conda-forge is required to be higher priority than defaults, then perhaps the tooling should enforce that situation? I dunno

BTW, we cannot say that b/c some situations may require the opposite or a single channel, etc... It will depend on what you want and from what channels you want. The only solution here is to empower users with docs and how multiple channels behave.

[fmaussion]

We are experiencing this on Travis currently: https://travis-ci.org/fmaussion/salem/jobs/407971313

Here is our environment.yml: https://github.com/fmaussion/salem/blob/master/ci/requirements-py36-all.yml

Any idea how to deal with this?

[ocefpaf]

https://github.com/conda-forge/fiona-feedstock/pull/85 may fix this but we need to wait for the new binaries to be up in the channel.

[fmaussion]

thanks @ocefpaf ! I just wanted to make sure that you guys are aware of it.

Same error at the time of writing, but I'll test it from time to time!

[ocefpaf]

@fmaussion you can follow the development of this issue at https://github.com/conda-forge/gdal-feedstock/pull/214

[ocefpaf]

@fmaussion can you re-try that env? Things should be back to normal on Python 3.6.

I'm still working on Python 2.7 and 3.5 though.

[fmaussion]

@ocefpaf all green! https://travis-ci.org/fmaussion/salem/builds/408047670?utm_source=github_status&utm_medium=notification

Thanks so much for this

[Casyfill]

having same issue, didn't get how to fix it... fiona 1.7.11 / 1.7.13 gdal 2.2.4

[ocefpaf]

having same issue, didn't get how to fix it...

The new packages are up so I'm guessing you are facing:

https://conda-forge.org/docs/conda-forge_gotchas.html

Without details of you env we cannot debug it though. See https://github.com/conda-forge/fiona-feedstock/issues/77#issuecomment-392198150

[Casyfill]

here is the environment;
https://gist.github.com/Casyfill/c0066d56cd6e3ee086592afe2cbaacdf

[ocefpaf]

The tight pins there are troublesome and I bet conda is doing some gymnastics to be able to get all the packages with those versions, including mixing incompatible packages from different channels.

PS: the name root can also be another source of trouble for you if you are using an old conda version (the new versions renamed root to base).

[Casyfill]

thanks, I will try unpinning and changing the name - although everything worked until the mids of last week

[ougx]

I am having the same issue.

https://github.com/conda-forge/fiona-feedstock/issues/77#issuecomment-409363655