search

conda-forge / geopandas-feedstock

A conda-smithy repository for geopandas.
BSD 3-Clause "New" or "Revised" License
16 stars 14 forks source link

broken proj database #72

Closed dazza-codes closed 4 years ago

dazza-codes commented 4 years ago

Issue:

Various libraries cannot find the proj.db file in a conda-env; since geopandas often dictates what the "compatible" libraries are, this issue is posted here because proj is the core of all the CRS operations and finding a set of "compatible" versions will always need to involve geopandas.

Create conda env with latest versions

export CONDA_ENV=gis

conda deactivate
conda create -y -n "${CONDA_ENV}" \
  --channel conda-forge \
  --override-channels \
  python=3.7

conda activate "${CONDA_ENV}"
conda config --env --add channels conda-forge
conda config --env --set channel_priority strict

conda install -y \
    click \
    dask distributed \
    requests \
    geopandas shapely rasterio rtree \
    gdal libgdal \
    pyproj \
    proj \
    proj-datumgrid-europe \
    proj-datumgrid-north-america \
    proj-datumgrid-oceania \
    proj-datumgrid-world \
    pyarrow fastparquet \
    numpy scipy xarray
$ gdal_polygonize.py input.tif -b 1 -q -f GPKG output.gpkg 
ERROR 1: PROJ: proj_create_from_database: Open of /opt/conda/envs/gis/share/proj failed
proj_create_from_database: Open of /opt/conda/envs/gis/share/proj failed


Environment (conda list):

``` $ conda list # packages in environment at /opt/conda/envs/gis: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 0_gnu conda-forge affine 2.3.0 py_0 conda-forge arrow-cpp 0.15.1 py37h982ac2c_6 conda-forge attrs 19.3.0 py_0 conda-forge awscli 1.17.8 pypi_0 pypi bokeh 1.4.0 py37_0 conda-forge boost-cpp 1.70.0 h8e57a91_2 conda-forge boto3 1.11.8 pypi_0 pypi botocore 1.14.8 pypi_0 pypi brotli 1.0.7 he1b5a44_1000 conda-forge bzip2 1.0.8 h516909a_2 conda-forge c-ares 1.15.0 h516909a_1001 conda-forge ca-certificates 2019.11.28 hecc5488_0 conda-forge cairo 1.16.0 hfb77d84_1002 conda-forge certifi 2019.11.28 py37_0 conda-forge cffi 1.13.2 py37h8022711_0 conda-forge cfitsio 3.470 hb60a0a2_2 conda-forge chardet 3.0.4 py37_1003 conda-forge click 7.0 py_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.5.0 py_0 conda-forge cloudpickle 1.2.2 py_1 conda-forge colorama 0.4.1 pypi_0 pypi cryptography 2.8 py37h72c5cf5_1 conda-forge curl 7.65.3 hf8cf82a_0 conda-forge cytoolz 0.10.1 py37h516909a_0 conda-forge dask 2.9.2 py_0 conda-forge dask-core 2.9.2 py_0 conda-forge dataclasses 0.6 pypi_0 pypi distributed 2.9.3 py_0 conda-forge docutils 0.15.2 pypi_0 pypi double-conversion 3.1.5 he1b5a44_2 conda-forge expat 2.2.5 he1b5a44_1004 conda-forge fastparquet 0.3.2 py37hc1659b7_0 conda-forge fiona 1.8.13 py37h900e953_0 conda-forge fontconfig 2.13.1 h86ecdb6_1001 conda-forge freetype 2.10.0 he983fc9_1 conda-forge freexl 1.0.5 h14c3975_1002 conda-forge fsspec 0.6.2 py_0 conda-forge gdal 3.0.3 py37hbb6b9fb_0 conda-forge geopandas 0.6.2 py_0 conda-forge geos 3.8.0 he1b5a44_0 conda-forge geotiff 1.5.1 h38872f0_8 conda-forge gettext 0.19.8.1 hc5be6a0_1002 conda-forge gflags 2.2.2 he1b5a44_1002 conda-forge giflib 5.2.1 h516909a_1 conda-forge glib 2.58.3 py37h6f030ca_1002 conda-forge glog 0.4.0 he1b5a44_1 conda-forge grpc-cpp 1.25.0 h213be95_2 conda-forge hdf4 4.2.13 hf30be14_1003 conda-forge hdf5 1.10.5 nompi_h3c11f04_1104 conda-forge heapdict 1.0.1 py_0 conda-forge icu 64.2 he1b5a44_1 conda-forge idna 2.8 py37_1000 conda-forge jinja2 2.10.3 py_0 conda-forge jpeg 9c h14c3975_1001 conda-forge json-c 0.13.1 h14c3975_1001 conda-forge kealib 1.4.10 h58c409b_1005 conda-forge krb5 1.16.4 h2fd8d38_0 conda-forge ld_impl_linux-64 2.33.1 h53a641e_7 conda-forge libblas 3.8.0 14_openblas conda-forge libcblas 3.8.0 14_openblas conda-forge libcurl 7.65.3 hda55be3_0 conda-forge libdap4 3.20.4 hd3bb157_0 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libevent 2.1.10 h72c5cf5_0 conda-forge libffi 3.2.1 he1b5a44_1006 conda-forge libgcc-ng 9.2.0 h24d8f2e_2 conda-forge libgdal 3.0.3 h20022a4_0 conda-forge libgfortran-ng 7.3.0 hdf63c60_4 conda-forge libgomp 9.2.0 h24d8f2e_2 conda-forge libiconv 1.15 h516909a_1005 conda-forge libkml 1.3.0 h4fcabce_1010 conda-forge liblapack 3.8.0 14_openblas conda-forge libllvm8 8.0.1 hc9558a2_0 conda-forge libnetcdf 4.7.3 nompi_h9f9fd6a_101 conda-forge libopenblas 0.3.7 h5ec1e0e_6 conda-forge libpng 1.6.37 hed695b0_0 conda-forge libpq 12.1 hd9ab2ff_0 conda-forge libprotobuf 3.11.0 h8b12597_0 conda-forge libspatialindex 1.9.3 he1b5a44_1 conda-forge libspatialite 4.3.0a ha48a99a_1034 conda-forge libssh2 1.8.2 h22169c7_2 conda-forge libstdcxx-ng 9.2.0 hdf63c60_2 conda-forge libtiff 4.1.0 hc3755c2_3 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libwebp 1.0.2 h56121f0_5 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxml2 2.9.10 hee79883_0 conda-forge llvmlite 0.31.0 py37h8b12597_0 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.8.3 he1b5a44_1001 conda-forge markupsafe 1.1.1 py37h516909a_0 conda-forge mercantile 1.1.2 pypi_0 pypi msgpack-python 0.6.2 py37hc9558a2_0 conda-forge munch 2.5.0 py_0 conda-forge ncurses 6.1 hf484d3e_1002 conda-forge numba 0.47.0 py37hb3f55d8_0 conda-forge numpy 1.17.5 py37h95a1406_0 conda-forge olefile 0.46 py_0 conda-forge openjpeg 2.3.1 h981e76c_3 conda-forge openssl 1.1.1d h516909a_0 conda-forge packaging 20.0 py_0 conda-forge pandas 0.25.3 py37hb3f55d8_0 conda-forge parquet-cpp 1.5.1 2 conda-forge parse 1.14.0 pypi_0 pypi partd 1.1.0 py_0 conda-forge pcre 8.43 he1b5a44_0 conda-forge pillow 7.0.0 py37hefe7db6_0 conda-forge pip 20.0.1 py37_0 conda-forge pixman 0.38.0 h516909a_1003 conda-forge poppler 0.67.0 h14e79db_8 conda-forge poppler-data 0.4.9 1 conda-forge postgresql 12.1 hc63931a_0 conda-forge proj 6.3.0 hc80f0dc_0 conda-forge proj-datumgrid-europe 1.4 3 conda-forge proj-datumgrid-north-america 1.2 3 conda-forge proj-datumgrid-oceania 1.0 3 conda-forge proj-datumgrid-world 1.0 3 conda-forge psutil 5.6.7 py37h516909a_0 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge pyarrow 0.15.1 py37h8b68381_1 conda-forge pyasn1 0.4.8 pypi_0 pypi pycparser 2.19 py37_1 conda-forge pyopenssl 19.1.0 py37_0 conda-forge pyparsing 2.4.6 py_0 conda-forge pyproj 2.4.2.post1 py37h8ff28aa_1 conda-forge pysocks 1.7.1 py37_0 conda-forge python 3.7.6 h357f687_2 conda-forge python-dateutil 2.8.1 py_0 conda-forge pytz 2019.3 py_0 conda-forge pyyaml 5.2 py37h516909a_0 conda-forge rasterio 1.1.1 py37h900e953_0 conda-forge re2 2020.01.01 he1b5a44_0 conda-forge readline 8.0 hf8c457e_0 conda-forge requests 2.22.0 py37_1 conda-forge rsa 3.4.2 pypi_0 pypi rtree 0.9.3 py37h7b0cdae_0 conda-forge s3fs 0.2.2 pypi_0 pypi s3transfer 0.3.2 pypi_0 pypi scipy 1.4.1 py37h921218d_0 conda-forge setuptools 45.1.0 py37_0 conda-forge shapely 1.6.4 py37h5d51c17_1007 conda-forge six 1.14.0 py37_0 conda-forge snappy 1.1.7 he1b5a44_1003 conda-forge snuggs 1.4.7 py_0 conda-forge sortedcontainers 2.1.0 py_0 conda-forge sqlite 3.30.1 hcee41ef_0 conda-forge tbb 2018.0.5 h2d50403_0 conda-forge tblib 1.6.0 py_0 conda-forge thrift 0.11.0 py37he1b5a44_1001 conda-forge thrift-cpp 0.12.0 hf3afdfd_1004 conda-forge tiledb 1.7.0 hcde45ca_2 conda-forge tk 8.6.10 hed695b0_0 conda-forge toolz 0.10.0 py_0 conda-forge tornado 6.0.3 py37h516909a_0 conda-forge tzcode 2019a h516909a_1002 conda-forge uriparser 0.9.3 he1b5a44_1 conda-forge urllib3 1.25.7 py37_0 conda-forge wheel 0.33.6 py37_0 conda-forge xarray 0.14.1 py_1 conda-forge xerces-c 3.2.2 h8412b87_1004 conda-forge xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.9 h516909a_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxrender 0.9.10 h516909a_1002 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.4 h14c3975_1001 conda-forge yaml 0.2.2 h516909a_1 conda-forge zict 1.0.0 py_0 conda-forge zlib 1.2.11 h516909a_1006 conda-forge zstd 1.4.4 h3b9ef0a_1 conda-forge ```


Details about conda and system ( conda info ):

``` $ conda info active environment : gis active env location : /opt/conda/envs/gis shell level : 1 user config file : /home/joe/.condarc populated config files : /home/joe/.condarc /opt/conda/envs/gis/.condarc conda version : 4.8.1 conda-build version : not installed python version : 3.7.4.final.0 virtual packages : __glibc=2.27 base environment : /opt/conda (writable) channel URLs : https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /opt/conda/pkgs /home/dlweber/.conda/pkgs envs directories : /opt/conda/envs /home/dlweber/.conda/envs platform : linux-64 user-agent : conda/4.8.1 requests/2.22.0 CPython/3.7.4 Linux/4.15.0-74-generic ubuntu/18.04.3 glibc/2.27 UID:GID : 1000:1000 netrc file : None offline mode : False ```
dazza-codes commented 4 years ago

Don't take this as authoritative commentary - it's learned from trial-n-error - part of the problem seems to be that conda-forge has renamed proj4<=6.1 to proj>6.1 and somewhere along the way the proj>6.1 project has split out the code project from the proj.db database project, so if you only install proj>6.1 without the additional proj-datumgrid stuff, then all the libs that look for CRS data can't find it. What makes a few things more complex is that often when working with CRS, it's used with GeoTIFF data files, so often the entire GIS/CRS ecosystem needs some combinations of geopandas, pyproj (with proj4<=6.1 and proj>6.1 + proj-datumgrid*), gdal (with fiona and rasterio), with the underlying pandas and numpy libs. From this perspective, it's difficult to keep track of all the library versions that cooperate well together. It might help if the conda-forge project had some kind of packaging process that was similar to Debian packaging, where some top-level integrations try to identify and maintain some high-level packages that effectively perform integration tests on an entire GIS/CRS ecosystem.

ocefpaf commented 4 years ago

I don't think that is the problem. Can you check what you have set as PROJ_LIB?

echo $PROJ_LIB

I was unable to reproduce your issue with a fresh env and I get virtually the same packages.

dazza-codes commented 4 years ago

I also cannot reproduce it reliably. IIRC, the PROJ_LIB was not set explicitly (and should not need to be set???). If it was set and pointing to the wrong location, that could explain it. (I guess my work has moved on and hopefully the 👻 is gone.)

ocefpaf commented 4 years ago

the PROJ_LIB was not set explicitly (and should not need to be set???).

It is automatically set by conda. Closing this b/c we don't have enough info to reproduce it.