search

Toblerity / Fiona

Fiona reads and writes geographic data files
https://fiona.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1.16k stars 203 forks source link

Error (GDAL) while importing fiona in Python3 (Ubuntu 18.04) #897

Open swiss-knight opened 4 years ago

swiss-knight commented 4 years ago

I met the following error while importing Fiona in Python3 on Ubuntu 18.04 after having freshly upgraded GDAL to its version 3.1.0:

>>> import fiona
ERROR 4: Unable to open EPSG support file gcs.csv. 
         Try setting the GDAL_DATA environment variable to point 
         to the directory containing EPSG csv files.

Further details here:
https://github.com/OSGeo/gdal/issues/2500 https://github.com/geopandas/geopandas/issues/1408

It seems that both fiona and rasterio keep installing this gcs.csv file which is no more needed by GDAL apparently in /usr/local/lib/python3.6/dist-packages/fiona/gdal_data/ and /usr/local/lib/python3.6/dist-packages/rasterio/gdal_data/, both installed with sudo -H pip3 install -U <package-name>.

>>> import fiona
ERROR 4: Unable to open EPSG support file gcs.csv.  
         Try setting the GDAL_DATA environment variable to point 
         to the directory containing EPSG csv files.
>>>

The same error is raised when calling the system tool fio in a bash terminal: 

$ fio --help
ERROR 4: Unable to open EPSG support file gcs.csv.  
         Try setting the GDAL_DATA environment variable to point 
         to the directory containing EPSG csv files.
Usage: fio [OPTIONS] COMMAND [ARGS]...
(...)

But If I do import rasterio first, fiona stops complaining:

>>> import rasterio
>>> import fiona
>>>

Environment

OS: Ubuntu 18.04.

Python3: Python 3.6.9 (default, Apr 18 2020, 01:56:04)

gdalinfo --version: GDAL 3.1.0, released 2020/05/03

fiona:

>>> import fiona
>>> fiona.__version__
'1.8.13.post1'
>>> fiona.__file__
'/usr/local/lib/python3.6/dist-packages/fiona/__init__.py'

EDIT

I first installed gdal from apt long time ago, prior to every other things, which for Ubuntu 18.04 is: 

$ apt-cache policy gdal-bin
gdal-bin:
  Installed: 2.4.0+10-0bionic1
  Candidate: 2.4.0+10-0bionic1
  Version table:
 *** 2.4.0+10-0bionic1 100
        100 /var/lib/dpkg/status
     2.2.3+dfsg-2 500
        500 http://ch.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

I installed Fiona (and almost all my favorite geo spatial libraries) with pip (20.1), which is currently;
Fiona-1.8.13.post1-cp36-cp36m-manylinux1_x86_64.whl (14.7 MB)

and raterio:
rasterio-1.1.4-cp36-cp36m-manylinux1_x86_64.whl (18.2 MB)

I usually do it like:
sudo -H pip3 install -U <package-name> (-U for upgading to its latest available version).
That's why the packages installed this way go to /usr/local instead of /usr directly. Which is fine.

Then, yesterday, I faced some issues while inserting data in a PostgreSQL v.12 database using ogr2ogr. I figured out it was fixed in a recent commit so I decided to upgrade GDAL itself, from source, because the version from apt was too old. Therefore, I downloaded version 3.1.0 from here; https://github.com/OSGeo/gdal/releases/tag/v3.1.0 , compiled it, and installed it using checkinstall tool.

Now gdal version is:

$ gdalinfo --version
GDAL 3.1.0, released 2020/05/03

Then I upgade the python gdal package also using pip. This was the shipped wheel:
GDAL-3.1.0-cp36-cp36m-linux_x86_64.whl =>

>>> from osgeo import gdal
>>> gdal.__version__
'3.1.0'
>>> gdal.__file__
'/usr/local/lib/python3.6/dist-packages/osgeo/gdal.py'
>>>

I did not uninstall gdal 2.4 (the apt version) because many (many!) other softwares I also installed with apt, such as QGIS relies on it and would be dropped as well by apt!

After what I figured out the error message while loading geopandas, which brings me here aswell because loading fiona itself causes the same error. But strangely, if I load rasterio prior to fiona everything goes well. This is obscure to me.

Here is the last piece of information which was asked for:

>>> from fiona._env import get_gdal_release_name
>>> print(get_gdal_release_name())
2.4.4
>>>

These are also the 3 places I can find the file causing troubles:

$ find /usr -iname gcs.csv
/usr/share/gdal/2.4/gcs.csv
/usr/local/lib/python3.6/dist-packages/rasterio/gdal_data/gcs.csv
/usr/local/lib/python3.6/dist-packages/fiona/gdal_data/gcs.csv
rbuffat commented 4 years ago

@swiss-knight Could you provide information about how you installed your gdal, rasterio or Fiona? E.g. did you use the ubuntu packages (apt get install ...), or did you install Fiona using pip install or did you install everything from source?

If you search in Fiona's issue tracker, you will find some previously reported issues. Some of them indicate that this is related to broken installations of gdal and Fiona.

Some background information:

My guess is that importing rasterio sets the environment variable GDAL_DATA to a directory with an installation of gdal that is compatible with the one your installed Fiona expects.

You can check the gdal version Fiona uses with the following command:

>>> from fiona._env import get_gdal_release_name
>>> print(get_gdal_release_name())
swiss-knight commented 4 years ago

I edited the main question to keep important information at the same original place.

rbuffat commented 4 years ago

Both rasterio as well as fiona patch the environment variable GDAL_DATA if the variable is not already set (if they are installed as wheel). 

>>> import os
>>> print(os.getenv('GDAL_DATA', None))
None
>>> import rasterio
>>> print(os.getenv('GDAL_DATA', None))
/usr/lib/python3.8/site-packages/rasterio/gdal_data

Could you post the output of 

import os
print(os.getenv('GDAL_DATA', None))
import logging

logging.basicConfig(
    level=logging.DEBUG,
)

import fiona
swiss-knight commented 4 years ago

Yes, here you are:

1.

With rasterio loaded before fiona:

$ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> print(os.getenv('GDAL_DATA', None))
None
>>> import rasterio
>>> print(os.getenv('GDAL_DATA', None))
/usr/local/lib/python3.6/dist-packages/rasterio/gdal_data
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG,)
>>> import fiona
DEBUG:fiona.env:PROJ data files are available at built-in paths
DEBUG:fiona.env:Entering env context: <fiona.env.Env object at 0x7f468839be48>
DEBUG:fiona.env:Starting outermost env
DEBUG:fiona.env:No GDAL environment exists
DEBUG:fiona.env:New GDAL environment <fiona._env.GDALEnv object at 0x7f4690180da0> created
DEBUG:fiona._env:Logging error handler pushed.
DEBUG:fiona._env:All drivers registered.
DEBUG:fiona._env:GDAL_DATA found in environment: '/usr/local/lib/python3.6/dist-packages/rasterio/gdal_data'.
DEBUG:fiona._env:PROJ data files are available at built-in paths
DEBUG:fiona._env:Started GDALEnv <fiona._env.GDALEnv object at 0x7f4690180da0>.
DEBUG:fiona.env:Updated existing <fiona._env.GDALEnv object at 0x7f4690180da0> with options {}
DEBUG:fiona.env:Entered env context: <fiona.env.Env object at 0x7f468839be48>
DEBUG:fiona.env:Exiting env context: <fiona.env.Env object at 0x7f468839be48>
DEBUG:fiona.env:Cleared existing <fiona._env.GDALEnv object at 0x7f4690180da0> options
DEBUG:fiona._env:Stopping GDALEnv <fiona._env.GDALEnv object at 0x7f4690180da0>.
DEBUG:fiona._env:Error handler popped.
DEBUG:fiona._env:Stopped GDALEnv <fiona._env.GDALEnv object at 0x7f4690180da0>.
DEBUG:fiona.env:Exiting outermost env
DEBUG:fiona.env:Exited env context: <fiona.env.Env object at 0x7f468839be48>
>>>

2.

Without rasterio loaded (in a new terminal):

$ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> print(os.getenv('GDAL_DATA', None))
None
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG,)
>>> import fiona
DEBUG:fiona.env:GDAL data files are available at built-in paths
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
DEBUG:fiona.env:PROJ data not found in environment, setting to '/usr/local/lib/python3.6/dist-packages/fiona/proj_data'.
DEBUG:fiona.env:Entering env context: <fiona.env.Env object at 0x7f8b1a396eb8>
DEBUG:fiona.env:Starting outermost env
DEBUG:fiona.env:No GDAL environment exists
DEBUG:fiona.env:New GDAL environment <fiona._env.GDALEnv object at 0x7f8b1a396f28> created
DEBUG:fiona._env:Logging error handler pushed.
DEBUG:fiona._env:All drivers registered.
DEBUG:fiona._env:GDAL data files are available at built-in paths
DEBUG:fiona._env:PROJ_LIB found in environment: '/usr/local/lib/python3.6/dist-packages/fiona/proj_data'.
DEBUG:fiona._env:Started GDALEnv <fiona._env.GDALEnv object at 0x7f8b1a396f28>.
DEBUG:fiona.env:Updated existing <fiona._env.GDALEnv object at 0x7f8b1a396f28> with options {}
DEBUG:fiona.env:Entered env context: <fiona.env.Env object at 0x7f8b1a396eb8>
DEBUG:fiona.env:Exiting env context: <fiona.env.Env object at 0x7f8b1a396eb8>
DEBUG:fiona.env:Cleared existing <fiona._env.GDALEnv object at 0x7f8b1a396f28> options
DEBUG:fiona._env:Stopping GDALEnv <fiona._env.GDALEnv object at 0x7f8b1a396f28>.
DEBUG:fiona._env:Error handler popped.
DEBUG:fiona._env:Stopped GDALEnv <fiona._env.GDALEnv object at 0x7f8b1a396f28>.
DEBUG:fiona.env:Exiting outermost env
DEBUG:fiona.env:Exited env context: <fiona.env.Env object at 0x7f8b1a396eb8>
>>>
rbuffat commented 4 years ago

Did you change any environment variables while install gdal or proj?

What is probably happening here is that fiona finds your new gdal 3.1 installation while it is compiled against gdal 2.4.4. In contrast, rasterio finds the gdal version included in the wheel and sets GDAL_DATA to '/usr/local/lib/python3.6/dist-packages/rasterio/gdal_data', which also works with your fiona as it's wheel includes the same version of gdal. I'm not really sure why Fiona finds your newer version of gdal, while rasterio not, as their search strategy is quite similar. Without having an environment to reproduce this is probably tricky to debug.

As a workaround, setting the gdal data environment variable before your script should work: os.environ["GDAL_DATA"] = "/usr/local/lib/python3.6/dist-packages/fiona/gdal_data'"

I would suggest that you try to uninstall Fiona (pip uninstall Fiona, maybe you have to execute it multiple times to uninstall all installed versions of fiona) and reinstall Fiona (pip install fiona).

If this doesn't help, you could try to install fiona using the gdal of your system: pip install fiona --no-binary :all:. You need to have cython installed for that. However, as you have multiple version of gdal installed you need to be careful about how you set your PATH variable. I assume the first gdal-config that is in your PATH will be used. The question will then be if this is the same gdal CPLFindFile finds.

rbuffat commented 4 years ago

@swiss-knight thanks for all the work!

I could reproduce the issue in a fresh ubuntu VM. Rasterio searches first in the wheel directory while Fiona doesn't.

Temporarily, you could edit /usr/local/lib/python3.6/dist-packages/fiona/env.py and replace 

if "GDAL_DATA" not in os.environ:

    # See https://github.com/mapbox/rasterio/issues/1631.
    if GDALDataFinder().find_file("header.dxf"):
        log.debug("GDAL data files are available at built-in paths")

    else:
        path = GDALDataFinder().search()

        if path:
            os.environ['GDAL_DATA'] = path
            log.debug("GDAL_DATA not found in environment, set to %r.", path)

with 

if "GDAL_DATA" not in os.environ:

    path = GDALDataFinder().search_wheel()

    if path:
        os.environ['GDAL_DATA'] = path
        log.debug("GDAL data found in package, GDAL_DATA set to %r.", path)

    # See https://github.com/mapbox/rasterio/issues/1631.
    elif GDALDataFinder().find_file("header.dxf"):
        log.debug("GDAL data files are available at built-in paths")

    else:
        path = GDALDataFinder().search()

        if path:
            os.environ['GDAL_DATA'] = path
            log.debug("GDAL_DATA not found in environment, set to %r.", path)
swiss-knight commented 4 years ago

You're welcome, I'm glad to help when I can, especially for those geo-libraries that I use on a regular basis.

I usually don't tweak environment variable while installing libraries and softwares, I try to stick to the default guidelines or install instructions, so was the case for compiling GDAL and all my Python packages, which I simply installed using pip.

Thanks a lot for your workaround which seems to have fix the problem:

$ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import logging, os
>>> print(os.getenv('GDAL_DATA', None))
None
>>> logging.basicConfig(
...     level=logging.DEBUG,
... )
>>> import fiona
DEBUG:fiona.env:GDAL data found in package, GDAL_DATA set to '/usr/local/lib/python3.6/dist-packages/fiona/gdal_data'.
DEBUG:fiona.env:PROJ data files are available at built-in paths
DEBUG:fiona.env:Entering env context: <fiona.env.Env object at 0x7f10ec7c8f28>
DEBUG:fiona.env:Starting outermost env
DEBUG:fiona.env:No GDAL environment exists
DEBUG:fiona.env:New GDAL environment <fiona._env.GDALEnv object at 0x7f10ec7c8f98> created
DEBUG:fiona._env:Logging error handler pushed.
DEBUG:fiona._env:All drivers registered.
DEBUG:fiona._env:GDAL_DATA found in environment: '/usr/local/lib/python3.6/dist-packages/fiona/gdal_data'.
DEBUG:fiona._env:PROJ data files are available at built-in paths
DEBUG:fiona._env:Started GDALEnv <fiona._env.GDALEnv object at 0x7f10ec7c8f98>.
DEBUG:fiona.env:Updated existing <fiona._env.GDALEnv object at 0x7f10ec7c8f98> with options {}
DEBUG:fiona.env:Entered env context: <fiona.env.Env object at 0x7f10ec7c8f28>
DEBUG:fiona.env:Exiting env context: <fiona.env.Env object at 0x7f10ec7c8f28>
DEBUG:fiona.env:Cleared existing <fiona._env.GDALEnv object at 0x7f10ec7c8f98> options
DEBUG:fiona._env:Stopping GDALEnv <fiona._env.GDALEnv object at 0x7f10ec7c8f98>.
DEBUG:fiona._env:Error handler popped.
DEBUG:fiona._env:Stopped GDALEnv <fiona._env.GDALEnv object at 0x7f10ec7c8f98>.
DEBUG:fiona.env:Exiting outermost env
DEBUG:fiona.env:Exited env context: <fiona.env.Env object at 0x7f10ec7c8f28>
>>>
demiurg commented 4 years ago

Just upgraded my homebrew gdal 2.4 to 3.1.1 bottle release and hit the same exact issue. Same logging output.

rdmurphy commented 4 years ago

Also in a similar scenario as @demiurg (jumped from gdal 2 to gdal 3 via homebrew) and suddenly started getting this error.

rbuffat commented 4 years ago

@demiurg @rdmurphy The fix is already merged in maint-1.8 branch but not yet released. Until a new release is out, you could either manually apply the fix from https://github.com/Toblerity/Fiona/issues/897#issuecomment-626223187 or use the maint-1.8 branch.

jdmcbr commented 4 years ago

Thanks @swiss-knight and @rbuffat. I did a fresh GDAL 3.1.2 install this weekend and started hitting this, and was confused given I knew the gcs.csv was no longer present in GDAL as of 3.0. Installing from maint-1.8 solved it!

dotysan commented 3 months ago

This happened for me when I built Fiona against GDAL built with OGR_BUILD_OPTIONAL_DRIVERS=OFF. The solution was to tell GDAL to turn them back on OGR_ENABLE_DRIVER_DXF=ON.