Rasterio can't find proj/gdal

[adamjstewart]

• [x] I read the conda-forge documentation and could not find the solution for my problem there.

Issue: I'm trying to install rasterio on Windows via GitHub Actions so that I can test my software. However, the rasterio installation doesn't seem to work:

rasterio.errors.CRSError: The EPSG code is unknown. PROJ: proj_create_from_database: Cannot find proj.db

After reading the rasterio FAQ, it seems that the issue is that PROJ_LIB isn't set. Could this be done in the rasterio feedstock? It seems to be done in the conda-forge version of proj, but conda is choosing to use the defaults proj instead. If I explicitly force conda to only use conda-forge, I get a different error:

    from rasterio._base import gdal_version
E   ImportError: DLL load failed: The specified procedure could not be found.

This one I have no idea how to solve. The below conda list is from the first test (mix of conda-forge and defaults) but I can give the conda list for the second test (only conda-forge) if you want it.

Environment (conda list):

``` $ conda list # packages in environment at C:\Miniconda\envs\test: # # Name Version Build Channel affine 2.3.0 py_0 conda-forge attrs 21.2.0 pyhd8ed1ab_0 conda-forge bzip2 1.0.8 h8ffe710_4 conda-forge ca-certificates 2021.5.30 h5b45459_0 conda-forge certifi 2021.5.30 py36ha15d459_0 conda-forge cfitsio 3.470 h0af3d06_7 conda-forge click 7.1.2 pyh9f0ad1d_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.7.2 pyhd8ed1ab_0 conda-forge curl 7.78.0 h789b8ee_0 conda-forge expat 2.4.1 h39d44d4_0 conda-forge freexl 1.0.6 ha8e266a_0 conda-forge geos 3.8.0 h33f27b4_0 geotiff 1.6.0 h5770a2b_0 h5py 2.10.0 nompi_py36h654ec23_106 conda-forge hdf4 4.2.13 h712560f_2 hdf5 1.10.6 nompi_h5268f04_1114 conda-forge intel-openmp 2021.3.0 h57928b3_3372 conda-forge jbig 2.1 h8d14728_2003 conda-forge jpeg 9d h8ffe710_0 conda-forge kealib 1.4.14 h96bfa42_2 conda-forge krb5 1.19.2 hbae68bd_0 conda-forge lerc 2.2.1 h0e60522_0 conda-forge libblas 3.9.0 10_mkl conda-forge libcblas 3.9.0 10_mkl conda-forge libcurl 7.78.0 h789b8ee_0 conda-forge libdeflate 1.7 h8ffe710_5 conda-forge libgdal 3.0.2 ha1b3edf_1 libiconv 1.16 he774522_0 conda-forge liblapack 3.9.0 10_mkl conda-forge libnetcdf 4.6.1 hf59b723_4 libpng 1.6.37 h1d00b33_2 conda-forge libpq 12.2 hb652d5d_1 libspatialite 4.3.0a h7ffb84d_0 libssh2 1.9.0 h680486a_6 conda-forge libtiff 4.3.0 h0c97f57_1 conda-forge libxml2 2.9.12 hf5bbc77_0 conda-forge lz4-c 1.9.3 h8ffe710_0 conda-forge m2w64-expat 2.1.1 2 conda-forge m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gettext 0.19.7 2 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libiconv 1.14 6 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge m2w64-xz 5.2.2 2 conda-forge mkl 2021.3.0 hb70f87d_564 conda-forge msys2-conda-epoch 20160418 1 conda-forge numpy 1.19.5 py36h4b40d73_2 conda-forge openjpeg 2.4.0 hb211442_1 conda-forge openssl 1.1.1k h8ffe710_0 conda-forge pip 21.2.1 pyhd8ed1ab_0 conda-forge postgresql 12.2 hb652d5d_1 proj 6.2.1 h9f7ef89_0 pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyreadline 2.1 py36ha15d459_1003 conda-forge python 3.6.13 h39d44d4_0_cpython conda-forge python_abi 3.6 2_cp36m conda-forge rasterio 1.1.0 py36h039b02d_0 setuptools 49.6.0 py36ha15d459_3 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snuggs 1.4.7 py_0 conda-forge sqlite 3.36.0 h8ffe710_0 conda-forge tbb 2021.3.0 h2d74725_0 conda-forge tiledb 2.3.2 h78dabda_0 conda-forge tk 8.6.10 h8ffe710_1 conda-forge ucrt 10.0.20348.0 h57928b3_0 conda-forge vc 14.2 hb210afc_5 conda-forge vs2015_runtime 14.29.30037 h902a5da_5 conda-forge wheel 0.36.2 pyhd3deb0d_0 conda-forge wincertstore 0.2 py36ha15d459_1006 conda-forge xerces-c 3.2.3 h0e60522_2 conda-forge xz 5.2.5 h62dcd97_1 conda-forge zlib 1.2.11 h62dcd97_1010 conda-forge zstd 1.5.0 h6255e5f_0 conda-forge ```

Details about conda and system ( conda info ):

``` $ conda info active environment : test active env location : C:\Miniconda\envs\test shell level : 2 user config file : C:\Users\runneradmin\.condarc populated config files : C:\Users\runneradmin\.condarc conda version : 4.10.3 conda-build version : not installed python version : 3.9.5.final.0 virtual packages : __win=0=0 __archspec=1=x86_64 base environment : C:\Miniconda (writable) conda av data dir : C:\Miniconda\etc\conda conda av metadata url : None channel URLs : https://conda.anaconda.org/conda-forge/win-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch package cache : C:\Users\runneradmin\conda_pkgs_dir envs directories : C:\Miniconda\envs C:\Users\runneradmin\.conda\envs C:\Users\runneradmin\AppData\Local\conda\conda\envs platform : win-64 user-agent : conda/4.10.3 requests/2.25.1 CPython/3.9.5 Windows/10 Windows/10.0.17763 administrator : True netrc file : None offline mode : False ```

@calebrob6

[snowman2]

@adamjstewart it appears like you are pulling libgdal from defaults. What is in your .condarc file?

[adamjstewart]

I'm not sure how to access the .condarc file on GitHub Actions, but I'm just using whatever the default is. Note that the conda list above is when using both conda-forge and defaults. If I only use conda-forge, I get the second issue with GDAL.

[xylar]

@adamjstewart, I'm afraid these kinds of issues are pretty much inevitable if you mix and match compiled packages like gdal between channels. Please share the GDAL issue you're having with conda-forge on its own and we can try to help you work that one out. But issues with using defaults GDAL and conda-forge other packages are why conda-forge very strongly encourages you to run:

conda config --add channels conda-forge
conda config --set channel_priority strict

Before creating your environment. The DLLs across channels are simply not compatible with one another.

[snowman2]

See: https://github.com/corteva/rioxarray/pull/384

I added this before the install command:

conda config --add channels conda-forge
conda config --set channel_priority strict

[xylar]

@snowman2, jinx!

[ocefpaf]

[x] I read the conda-forge documentation and could not find the solution for my problem there.

I believe our docs jinxed you all first ;-p

@adamjstewart please take a look at those. They improved a lot lately and most of these questions are answered there and headaches avoided by following the instructions on them.

[adamjstewart]

Thanks for the quick feedback everyone! When using the following settings:

    - name: Install conda
      uses: conda-incubator/setup-miniconda@v2
      with:
        python-version: ${{ matrix.python-version }}
        channels: conda-forge
        channel-priority: strict
    - name: Install conda dependencies
      run: conda install h5py 'rasterio>=1.0'

I get the following environment:

Environment (conda list):

``` $ conda list # packages in environment at C:\Miniconda\envs\test: # # Name Version Build Channel affine 2.3.0 py_0 conda-forge attrs 21.2.0 pyhd8ed1ab_0 conda-forge boost-cpp 1.74.0 h5b4e17d_4 conda-forge bzip2 1.0.8 h8ffe710_4 conda-forge ca-certificates 2021.5.30 h5b45459_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cairo 1.16.0 hb19e0ff_1008 conda-forge certifi 2021.5.30 py36ha15d459_0 conda-forge cfitsio 3.470 h0af3d06_7 conda-forge click 7.1.2 pyh9f0ad1d_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.7.2 pyhd8ed1ab_0 conda-forge curl 7.78.0 h789b8ee_0 conda-forge expat 2.4.1 h39d44d4_0 conda-forge fontconfig 2.13.1 h1989441_1005 conda-forge freetype 2.10.4 h546665d_1 conda-forge freexl 1.0.6 ha8e266a_0 conda-forge geos 3.9.1 h39d44d4_2 conda-forge geotiff 1.6.0 ha8a8a2d_6 conda-forge gettext 0.19.8.1 h1a89ca6_1005 conda-forge h5py 3.1.0 nompi_py36hf359dfe_100 conda-forge hdf4 4.2.15 h0e5069d_3 conda-forge hdf5 1.10.6 nompi_h5268f04_1114 conda-forge icu 68.1 h0e60522_0 conda-forge intel-openmp 2021.3.0 h57928b3_3372 conda-forge jbig 2.1 h8d14728_2003 conda-forge jpeg 9d h8ffe710_0 conda-forge kealib 1.4.14 h96bfa42_2 conda-forge krb5 1.19.2 hbae68bd_0 conda-forge lerc 2.2.1 h0e60522_0 conda-forge libblas 3.9.0 10_mkl conda-forge libcblas 3.9.0 10_mkl conda-forge libcurl 7.78.0 h789b8ee_0 conda-forge libdeflate 1.7 h8ffe710_5 conda-forge libffi 3.3 h0e60522_2 conda-forge libgdal 3.1.4 ha6cada3_17 conda-forge libglib 2.68.3 h1e62bf3_0 conda-forge libiconv 1.16 he774522_0 conda-forge libkml 1.3.0 h9859afa_1014 conda-forge liblapack 3.9.0 10_mkl conda-forge libnetcdf 4.8.0 nompi_hf689e7d_103 conda-forge libpng 1.6.37 h1d00b33_2 conda-forge libpq 13.3 hfcc5ef8_0 conda-forge librttopo 1.1.0 hb340de5_6 conda-forge libspatialite 5.0.1 h762a7f4_5 conda-forge libssh2 1.9.0 h680486a_6 conda-forge libtiff 4.3.0 h0c97f57_1 conda-forge libwebp-base 1.2.0 h8ffe710_2 conda-forge libxml2 2.9.12 hf5bbc77_0 conda-forge libzip 1.8.0 hfed4ece_0 conda-forge lz4-c 1.9.3 h8ffe710_0 conda-forge mkl 2021.3.0 hb70f87d_564 conda-forge numpy 1.19.5 py36h4b40d73_2 conda-forge openjpeg 2.4.0 hb211442_1 conda-forge openssl 1.1.1k h8ffe710_0 conda-forge pcre 8.45 h0e60522_0 conda-forge pip 21.2.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h8ffe710_0 conda-forge poppler 21.03.0 h9ff6ed8_0 conda-forge poppler-data 0.4.10 0 conda-forge postgresql 13.3 h1c22c4f_0 conda-forge proj 8.0.1 h1cfcee9_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyreadline 2.1 py36ha15d459_1003 conda-forge python 3.6.13 h39d44d4_0_cpython conda-forge python_abi 3.6 2_cp36m conda-forge rasterio 1.1.7 py36hc1acebe_0 conda-forge setuptools 49.6.0 py36ha15d459_3 conda-forge snuggs 1.4.7 py_0 conda-forge sqlite 3.36.0 h8ffe710_0 conda-forge tbb 2021.3.0 h2d74725_0 conda-forge tiledb 2.3.2 h78dabda_0 conda-forge tk 8.6.10 h8ffe710_1 conda-forge ucrt 10.0.20348.0 h57928b3_0 conda-forge vc 14.2 hb210afc_5 conda-forge vs2015_runtime 14.29.30037 h902a5da_5 conda-forge wheel 0.36.2 pyhd3deb0d_0 conda-forge wincertstore 0.2 py36ha15d459_1006 conda-forge xerces-c 3.2.3 h0e60522_2 conda-forge xz 5.2.5 h62dcd97_1 conda-forge zlib 1.2.11 h62dcd97_1010 conda-forge zstd 1.5.0 h6255e5f_0 conda-forge ```

Again, the error message for this environment is:

    import rasterio
C:\Miniconda\envs\test\lib\site-packages\rasterio\__init__.py:22: in <module>
    from rasterio._base import gdal_version
E   ImportError: DLL load failed: The specified procedure could not be found.

[ocefpaf]

Python 3.6 is no longer supported by the scientific stack. Conda-forge will be dropping it soon but we already dropped it in some places, specially on Windows where it is harder to maintain.

[xylar]

Oh, good catch @ocefpaf! I hadn't noticed the python version. I wonder why the solver didn't just fail. But I guess there must be some combination of older packages that somehow works (in the sense that the solver is satisfied) but with a gdal that isn't, in fact, compatible.

@adamjstewart, I'm sure that's a frustrating situation. But if you are able to specify python >=3.7 as a constraint and try again, you might have better luck. I'm guessing you're getting python 3.6 because your Miniconda3 installation that you're starting from is pretty old and defaults to 3.6. Could that be the case?

[adamjstewart]

Python 3.6 is no longer supported by the scientific stack.

Thanks @ocefpaf! I just assumed this issue had nothing to do with Python version, but if I drop 3.6 from my testing matrix the rest of the tests pass!

I'm guessing you're getting python 3.6 because your Miniconda3 installation that you're starting from is pretty old and defaults to 3.6. Could that be the case?

I was explicitly testing and supporting Python 3.6, 3.7, 3.8, and 3.9 in my project, but it sounds like I'll be dropping 3.6 support now. It's unfortunate that the scientific stack is dropping support for versions of Python before they reach EOL, but that's obviously outside of my control.

[xylar]

@adamjstewart, numpy recommended dropping python 3.6 support on Jun 23, 2020 (more than a year ago) in the "scientific python ecosystem" as you can read here: https://numpy.org/neps/nep-0029-deprecation_policy.html

My understanding is that it's really only practical to do a good job at support 3 python versions simultaneously. That's certainly been my experience.