Missing PROJ_LIB environment variable in pyproj>=3.4.0

[danschef]

Solution to issue cannot be found in the documentation.

• [X] I checked the documentation.

Issue

The pyproj>=3.4.0 builds (e.g., py310hb1338dc_2) do not set the PROJ_LIB environment variable. This breaks my code when running any kind of reprojection (e.g., from gdalwarp) and raises the following error:

ERROR 1: PROJ: proj_create_from_database: Open of /path/to/myenv/share/proj failed

pyproj<3.4.0:

$ mamba create -n prj33 'pyproj<3.4.0' 'python=3.10'  # installs pyproj 3.3.1 py310hf94497c_1
$ mamba activate prj33
$ echo $PROJ_LIB
/home/me/mambaforge/envs/prj33/share/proj

pyproj>=3.4.0:

$ mamba create -n prj340 'pyproj=3.4.0' 'python=3.10'  # installs pyproj 3.4.0 py310hb1338dc_2
$ mamba activate prj340
$ echo $PROJ_LIB
-> not set

Installed packages

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libcurl                   7.87.0               hdc1c0ab_0    conda-forge
libdeflate                1.14                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libnghttp2                1.51.0               hff17c54_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libssh2                   1.10.0               hf14f497_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libtiff                   4.5.0                h82bc61c_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
openssl                   3.0.7                h0b41bf4_1    conda-forge
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
proj                      9.1.0                h8ffa02c_1    conda-forge
pyproj                    3.4.0           py310hb1338dc_2    conda-forge
python                    3.10.8          h4a9ceb5_0_cpython    conda-forge
python_abi                3.10                    3_cp310    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
setuptools                65.6.3             pyhd8ed1ab_0    conda-forge
sqlite                    3.40.0               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zstd                      1.5.2                h6239696_4    conda-forge

Environment info

active environment : prj340
    active env location : /home/me/mambaforge/envs/prj340
            shell level : 10
       user config file : /home/me/.condarc
 populated config files : /home/me/mambaforge/.condarc
                          /home/me/.condarc
          conda version : 22.11.1
    conda-build version : not installed
         python version : 3.10.8.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.31=0
                          __linux=5.4.0=0
                          __unix=0=0
       base environment : /home/me/mambaforge  (writable)
      conda av data dir : /home/me/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/me/mambaforge/pkgs
                          /home/me/.conda/pkgs
       envs directories : /home/me/mambaforge/envs
                          /home/me/.conda/envs
               platform : linux-64
             user-agent : conda/22.11.1 requests/2.28.1 CPython/3.10.8 Linux/5.4.0-52-generic ubuntu/20.04.5 glibc/2.31
                UID:GID : 1009:38755
             netrc file : None
           offline mode : False

[xylar]

@danschef, I'm sure this must be frustrating.

This issue relates to proj (https://github.com/conda-forge/proj.4-feedstock) rather than pyproj, since the variable is for the proj package's data.

In proj>=9.1.0, the environment variable is now $PROJ_DATA instead of $PROJ_LIB. It was changed in proj here: https://github.com/OSGeo/PROJ/pull/3253 and in proj.4-feedstock here: https://github.com/conda-forge/proj.4-feedstock/pull/125

I think if you're using $PROJ_LIB in your own code, you will now need to first check for $PROJ_DATA and then fall back on $PROJ_LIB.

[danschef]

Thanks, then this should probably respected in gdal, right? In the meanwhile, I can simply set PROJ_LIB with PROJ_DATA in my code.

[xylar]

@danschef, you're right that recent versions of gdal should presumably know about this change.

[xylar]

Could you provide a more detailed reproducer for the gdal error you're seeing and maybe post an issue on the gdal feedstock pointing it to this discussion?

[jorisvandenbossche]

We have been noticing a similar error in the geopandas docs (built on readthedocs), that we couldn't reproduce locally, and that might be related (https://github.com/geopandas/geopandas/issues/2683, https://github.com/geopandas/geopandas/issues/2652#issuecomment-1326668393, cc @ocefpaf)

I am wondering if it could somehow be related to the fact that released Fiona is not yet updated to work with PROJ_DATA instead of PROJ_LIB. Although when using conda, I would assume that the proj bindings in Fiona (through gdal?) will still find the data at the expected built-in path, even when PROJ_LIB is not set.

[jorisvandenbossche]

I could actually reproduce it by mimicking the output of readthedocs, by not activating the environment but using the full path to the python executable:

Create environment with released fiona (1.8.22) and latest gdal/proj

mamba create -n test-proj-data python=3.10 fiona

Run python by using full path (without activating environment):

$ /home/joris/miniconda3/envs/test-proj-data/bin/python
Python 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:26:04) [GCC 10.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fiona
ERROR 1: PROJ: proj_create_from_database: Open of /home/joris/miniconda3/envs/test-proj-data/share/proj failed
>>> import fiona.env
>>> fiona.env.PROJDataFinder().search()
'/home/joris/miniconda3/envs/test-proj-data/share/proj'

What is strange is that this directory is actually the correct directory that contains proj.db, so it is not clear to me why PROJ is raising this error.

Sorry for posting here in case it is not actually related. In any case, it is not related to pyproj in my case (but maybe @danschef for your actual use case where you encountered this, you also had other packages installed like fiona?)

[xylar]

@jorisvandenbossche, if you don't activate the environment, neither the $PROJ_LIB variable in the past nor the $PROJ_DATA` variable now will be set, so I don't see how this would have change recently.

[danschef]

I properly activated the environment, which sets $PROJ_DATA. But yes, I also had other libraries like fiona and gdal installed and importing fiona raises the error as in https://github.com/conda-forge/pyproj-feedstock/issues/130#issuecomment-1370918801.

To me, this looks like fiona needs to be updated to use $PROJ_DATA.

[xylar]

@danschef, that does seem like at least one likely source of the problem.

[snowman2]

Fiona PR https://github.com/Toblerity/Fiona/pull/1128

[jorisvandenbossche]

@xylar yes, I know that not activating the environment doesn't set the env variable (and myself I always work with activated envs), but it's just that it is for now the only way that I could reproduce this error locally. In a properly activated environment created with mamba create -n test-proj-data python=3.10 fiona, running import fiona does not print any error for me.

(and to be clear, I also doesn't actually "error", code actually runs fine, but PROJ for some reason prints an error about not being able to open the path)

@danschef do you also get the error with the mamba create example I gave? Or can you provide a full reproducer with output of the different steps?

[jorisvandenbossche]

Fiona PR Toblerity/Fiona#1128

Yes, but so that is only in 1.9.x, thus not yet released (maybe something to bring up with Fiona)

[snowman2]

Here is the 1.9 release issue: https://github.com/Toblerity/Fiona/issues/1053

[danschef]

Ok, it seems like the issue is somehow related to PyCharm. I just created an environment with mamba create -n fiona 'python=3.10' fiona (installs fiona 1.8.22 py310ha325b7b_5 + dependencies), activated it and on the conda command line I can import fiona without any issues. Also $PROJ_DATA is set correctly.

However, if I set this environment as the interpreter in PyCharm (2022.3.1), and import fiona on a Python console within PyCharm, I get ERROR 1: PROJ: proj_create_from_database: Open of /home/me/mambaforge/envs/fiona/share/proj failed and neither $PROJ_DATA nor $PROJ_LIB are set. This is strange because, e.g., $GDAL_DATA is properly set, so PyCharm seems to activate the environment as it should.

I can of course set $PROJ_DATA manually in PyCharm actually this should work out of the box.

[xylar]

@danschef, I realize this might not be resolved yet but it isn't ultimately a pyproj issue so I'm closing it. If the issue persists, I'd follow up on the fiona, proj.4 and any other affecte feedstocks.