Toblerity / Fiona

Fiona reads and writes geographic data files
https://fiona.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1.15k stars 202 forks source link

Circular imports for Python 3.8 on Windows if DLLs are already loaded #944

Closed rbuffat closed 4 years ago

rbuffat commented 4 years ago

WIth fiona 1.8.15, the issue is fixed now in the conda-forge builds, only the Windows py3.8 build is still failing: https://github.com/conda-forge/fiona-feedstock/pull/167

The failure is with the general error message of partially initialized module, so doesn't give much pointers (at least to me):

running run_test.py
Traceback (most recent call last):
  File "D:\bld\fiona_1599200296823\test_tmp\run_test.py", line 6, in <module>
    import fiona
  File "D:\bld\fiona_1599200296823\_test_env\lib\site-packages\fiona\__init__.py", line 84, in <module>
    import fiona._loading
  File "D:\bld\fiona_1599200296823\_test_env\lib\site-packages\fiona\_loading.py", line 44, in <module>
    import fiona.ogrext
  File "fiona/ogrext.pyx", line 29, in init fiona.ogrext
  File "D:\bld\fiona_1599200296823\_test_env\lib\site-packages\fiona\env.py", line 14, in <module>
    with fiona._loading.add_gdal_dll_directories():
AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)

Originally posted by @jorisvandenbossche in https://github.com/Toblerity/Fiona/issues/941#issuecomment-686982973

rbuffat commented 4 years ago

The issue is with https://github.com/Toblerity/Fiona/blob/52ad006172bdb0d9c240bd827b958a58c63e5303/fiona/_loading.py#L43-L46

Starting with Python 3.8 dll directories must be added with add_dll_directory() on Windows. (https://github.com/Toblerity/Fiona/issues/851)

On Fionas appveyor, the import of import fiona.ogrext will fail as there are no DLL presents at the time of import, or fiona.ogrext is already loaded. But in the case of conda, the DLLs are already loaded. Thus the import of fiona.ogrext does not raise an ImportError and fiona.ogrext will import fiona.env, that again imports fiona._loading, which leads to a circular import.

A quick fix could be to change the test import to something that does not import anything else from Fiona. E.g.

 try: 
     import fiona._err 
 except ImportError as e: 
     search_gdal_dll_directories() 

But this will be quite dangerous because with adding additional imports the problem can arise again.

sgillies commented 4 years ago

@rbuffat as we solve this, let's also move this code

if platform.system() == 'Windows' and sys.version_info >= (3, 8):
    # if loading of extension modules fails, search for gdal dll directories
    try:
        import fiona.ogrext
    except ImportError as e:
        search_gdal_dll_directories()

into add_gdal_dll_directories(). Code that runs on import only is difficult to test.

rbuffat commented 4 years ago

I created a PR in https://github.com/Toblerity/Fiona/pull/945. If you have already implemented something feel free to close it.

rbuffat commented 4 years ago

Fixed in Fiona 1.8.16

iboates commented 3 years ago

I'm leaving a comment here because it's the only Google result which matched my error exactly. Hopefully it can save someone else a similar headache.

I am using a conda environment when getting this error and it was solved by starting a new environment where I made sure that every package was being taken from the conda-forge channel (except for some libraries only available from the esri channel.

janhurst commented 3 years ago

I'm leaving a comment here because it's the only Google result which matched my error exactly. Hopefully it can save someone else a similar headache.

I am using a conda environment when getting this error and it was solved by starting a new environment where I made sure that every package was being taken from the conda-forge channel (except for some libraries only available from the esri channel.

I am seeing the same behaviour with conda 4.9.2 on Windows 10 trying to get geopandas up and running. Creating an environment and forcing only conda-forge packages. As best as I can tell everything is "correctly" coming from conda-forge.

various versions are: gdal 3.2.2 fiona 1.8.18 geopandas 0.9.0

jorisvandenbossche commented 3 years ago

@janhurst can you post the output of conda list in the environment?

jjohnarios commented 3 years ago

@jorisvandenbossche I'm getting the same error with the same conda and packages versions as @janhurst . I have some non conda-forge packages which i believe have nothing to do with geopandas. I'm posting the exact error i'm getting and my conda list output.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-a62d01c1d62e> in <module>
----> 1 import geopandas as gpd

~\Anaconda3\envs\nilmtk-env\lib\site-packages\geopandas\__init__.py in <module>
      5 from geopandas.array import points_from_xy  # noqa
      6 
----> 7 from geopandas.io.file import _read_file as read_file  # noqa
      8 from geopandas.io.arrow import _read_parquet as read_parquet  # noqa
      9 from geopandas.io.arrow import _read_feather as read_feather  # noqa

~\Anaconda3\envs\nilmtk-env\lib\site-packages\geopandas\io\file.py in <module>
     18 
     19 try:
---> 20     from fiona import Env as fiona_env
     21 except ImportError:
     22     try:

~\Anaconda3\envs\nilmtk-env\lib\site-packages\fiona\__init__.py in <module>
     83 
     84 import fiona._loading
---> 85 with fiona._loading.add_gdal_dll_directories():
     86     from fiona.collection import BytesCollection, Collection
     87     from fiona.drvsupport import supported_drivers

AttributeError: module 'fiona' has no attribute '_loading'

conda_list.txt

janhurst commented 3 years ago

I noticed that the environment seems to work ok when i try to read a file in a notebook when using jupyter lab, yet the same notebook will throw the error I posted when running inside vscode.

@jjohnarios do you happen to be using vscode?

I dug a bit further. If I start vscode from a Powershell prompt (eg code . and note using the base conda environment), then open a notebook, and set the notebook kernel to my Python interpreter I am seeing the issue. I confirmed that sys.version and sys.executable seem to be pointing to the "geo" conda environment correctly, just the import geopandas call failing as indicated above.

However, If I first activate the "geo" environment (eg conda activate geo) in Powershell, and then do the same process of starting vscode and set the notebook kernel etc the error does not occur!

I'm pretty confused now :) ... I don't know if this is a vscode issue, or perhaps something screwy in conda and its Powershell integration etc.

@jorisvandenbossche for reference my conda list output is attached, as well as the environment yaml.

geo.yml.txt conda_list.txt

jjohnarios commented 3 years ago

@janhurst I've tried vscode and import works fine. I was using jupyter notebook and spyder and the error kept occurring there. I made sure i was using the same kernel and environment. Consequently, i realised that : import geopandas works for me in python Anaconda prompt, in VScode, in Sublime Text and NOT IN jupyter notebook and spyder.

andrewnell commented 3 years ago

I have the exact same issue, where import geopandas works for me in the anaconda prompt and sublime, however, I get the same error as above in Jupyter Notebook, Jupyter Lab or Spyder.

It should also be noted that this is only for an environment I have created using Python 3.6, however, if I create an environment with Python 3.8 I do not have this issue. @jorisvandenbossche any ideas what may be causing this issue?

rbuffat commented 3 years ago

@jorisvandenbossche do you have an idea how to debug this? I don't have access to a Windows machine at the moment, so the only possibility would be to create a CI workflow that would reproduce this. But for that, we would require the exact steps to reproduce this issue.

Maybe placing print(dir(fiona)) in front of from fiona import Env as fiona_env in https://github.com/geopandas/geopandas/blob/master/geopandas/io/file.py#L20 might reveal some more useful information.

jorisvandenbossche commented 3 years ago

I also don't have a Windows machine readily available.
Since the issue seem to be triggered by using an environment that is not directly activated, but used as a kernel from some UI (notebook, spyder), one possible issue could be that fiona is not finding/looking for the correct GDAL location (although I still find it a strange error message for that ..).

@janhurst @jjohnarios @andrewnell can you try if setting the following environment variable before importing geopandas/fiona inside the notebook / spyder helps? It should be the equivalent of:

@set "GDAL_DATA=%CONDA_PREFIX%\Library\share\gdal"

so in terms of python code you can run in the notebook before importing geopandas/fiona, I think it would be something like

import os
os.environ["GDAL_DATA"] = os.environ["CONDA_PREFIX"] + "\Library\share\gdal"
jjohnarios commented 3 years ago

I also don't have a Windows machine readily available. Since the issue seem to be triggered by using an environment that is not directly activated, but used as a kernel from some UI (notebook, spyder), one possible issue could be that fiona is not finding/looking for the correct GDAL location (although I still find it a strange error message for that ..).

@janhurst @jjohnarios @andrewnell can you try if setting the following environment variable before importing geopandas/fiona inside the notebook / spyder helps? It should be the equivalent of:

@set "GDAL_DATA=%CONDA_PREFIX%\Library\share\gdal"

so in terms of python code you can run in the notebook before importing geopandas/fiona, I think it would be something like

import os
os.environ["GDAL_DATA"] = os.environ["CONDA_PREFIX"] + "\Library\share\gdal"

I've tried that but doesn't seem to work. Same error message.

jorisvandenbossche commented 3 years ago

@jjohnarios OK, thanks for trying!

Aonther question: you posted above your conda list output, and indeed I don't directly see anything suspicious (the packages no from conda-forge but from pypi shouldn't have an impact on fiona). But could you try if you can reproduce it with a minimal, newly created environment as well? Eg like the sequence of commands at https://geopandas.readthedocs.io/en/latest/getting_started/install.html#creating-a-new-environment

janhurst commented 3 years ago

Also didn't see any difference setting GDAL_DATA.

jorisvandenbossche commented 3 years ago

Another question (sorry for the many questions, but that's at the moment the only way for me to help debug it): do you only see it with the latest fiona 1.8.18? If you explicitly install fiona 1.8.17 in the environment, do you still see the error?

Or if you explicitly ask for an older GDAL? (you both seem to have 3.2.2 installed, which is relatively new (which might explain that it's a new issue)) Do you also have the error when creating the environment with fiona=1.8.18 gdal=3.1?

janhurst commented 3 years ago

But could you try if you can reproduce it with a minimal, newly created environment as well? Eg like the sequence of commands at https://geopandas.readthedocs.io/en/latest/getting_started/install.html#creating-a-new-environment

I tried to follow the getting started linked as exactly as possible. The only addition is that the ipykernel package is needed to actually switch to the kernel in vscode's notebook environment.

Unfortunately I am still seeing the same behaviour - starting vscode from the base environment and trying to select the geo_env kernel results in the module 'fiona' has no attribute '_loading' message in an AttributeError exception.

For reference the conda list output for the environment as described in the geopandas getting started plus conda install ipykernel is attached. geo_env_conda_list.txt

janhurst commented 3 years ago

Do you also have the error when creating the environment with fiona=1.8.18 gdal=3.1?

Missed this, its late here but I'll give this a shot in the next day or two.

RicardoSQueen commented 3 years ago

Hey, there,

Having the same issue as @janhurst.

Another question (sorry for the many questions, but that's at the moment the only way for me to help debug it): do you only see it with the latest fiona 1.8.18? If you explicitly install fiona 1.8.17 in the environment, do you still see the error?

Tried with fiona 1.8.17 and 1.8.16 and still no success (same error)

Or if you explicitly ask for an older GDAL? (you both seem to have 3.2.2 installed, which is relatively new (which might explain that it's a new issue)) Do you also have the error when creating the environment with fiona=1.8.18 gdal=3.1?

Tried that too and still no success. Here is my current conda list output: conda_list.txt

Just reforcing: on anaconda prompt the import occurs successfully, on vscode it does not work.

fargiolas commented 3 years ago

Probably a different bug, but I'm seeing a similar circular import error when I import geopandas (fiona 1.18.18, geopandas 0.9.0, gdal 3.2.2, installed with pip and brew on macos). The error goes away if I import fiona before geopandas.

jorisvandenbossche commented 3 years ago

OK, still a mystery .. (and thanks all for the testing!) Some other things to verify / try:

1. Does this only happen with geopandas 0.9, or also with older versions (0.8)?

2. Something that changed in the latest geopandas version, is that we delayed the fiona import (https://github.com/geopandas/geopandas/pull/1775), but which might have as concrete effect that the order of imports of the different geo-reletad packages changed.
So maybe something to try, does one of the following import sequences also trigger the error? (in the notebook/spyder/vscode environment)

import pyproj
import fiona
import shapely
import fiona

3. Another possible "pattern" in the posted conda list outputs, is that you all have pyproj 3.0.1. Could someone try creating an environment but with an older version of pyproj? (3.0.0 or 2.6.1)? (not directly an idea how that would influence it, but you never know ... and to eliminate possibilities)

ogelin commented 3 years ago

Hi, on MacOS using pip installs I also had this error when importing geopandas. Python 3.9.2.

I changed pyproj from 3.0.1 to 3.0.0 and now the error goes away as long as I import fiona before geopandas.

If I only import geopandas without fiona first, I still get the "Circular import" error message that others reported.

LSgeo commented 3 years ago

I noticed that the environment seems to work ok when I try to read a file in a notebook when using jupyter lab, yet the same notebook will throw the error I posted when running inside vscode.

@jjohnarios do you happen to be using vscode?

I dug a bit further. If I start vscode from a Powershell prompt (eg code . and note using the base conda environment), then open a notebook, and set the notebook kernel to my Python interpreter I am seeing the issue. I confirmed that sys.version and sys.executable seem to be pointing to the "geo" conda environment correctly, just the import geopandas call failing as indicated above.

However, If I first activate the "geo" environment (eg conda activate geo) in Powershell, and then do the same process of starting vscode and set the notebook kernel etc the error does not occur!

I'm pretty confused now :) ... I don't know if this is a vscode issue, or perhaps something screwy in conda and its Powershell integration etc.

@jorisvandenbossche for reference my conda list output is attached, as well as the environment yaml.

geo.yml.txt conda_list.txt

Hi, I've stumbled across here too, and wanted to add my two cents. I'm running a script in VScode on windows 10, and only encounter the issue when executing via a Jupyter Interactive notebook session.

I've encountered a lot of other dll issues with other libraries in the Jupyter Interactive window, that don't occur when running the script normally via the command line. There are a lot (of vaguely similar problems) spread across various github issues, but e.g. , vscode-jupyter and numpy

As you mention in the above comment, activating an environment and calling code to start vscode works around the issue for me. I'd love to learn why this is, and if there is a better solution.

jorisvandenbossche commented 3 years ago

Some general background context: conda has the concept of "environment activation" (which happens if you manually activate an environment in the console with conda activate env_name). This step sets up the environment to work properly, and for example sets the appropriate environment variables (eg it adds the env directory and the env's /bin directory to the PATH).

Specifically for fiona, we rely on GDAL, and the libgdal package from conda-forge has a specific activation script (see https://github.com/conda-forge/gdal-feedstock/tree/master/recipe/scripts), which sets the GDAL_DATA environment variable. In addition, I think fiona relies on being able to find the GDAL binary based on PATH.

If those environment variables are not properly set, packages will have problems finding the correct binaries / libraries.

And I suppose this is the problem that causes the issues you all encounter when using jupyter notebook or vscode.

For Jupyter Notebook with using kernels based on conda environments, this is a known problem that this usage pattern doesn't automatically activate the environment when starting the kernel (https://github.com/jupyter/notebook/issues/3588). One workaround is to install notebook in the environment you want to use, first activate the environment, and then start the notebook from that environment (this defeats the purpose of having multiple kernels a bit, though). Another solution is to use a tool like https://github.com/Anaconda-Platform/nb_conda_kernels that ensures kernels are activated properly. Finally, you could also prepend the appropriate ..\\envs\\env_name\\Library\\bin to your PATH manually in the notebook before importing fiona/geopandas.

For VSCode, I have less experience with this, but according to https://code.visualstudio.com/docs/python/environments#_select-and-activate-an-environment it seems that it should be possible to adjust your settings to ensure the environment is activated.


Now, what I am less sure about is why it seems that there now new issues with this (given that multiple people posted here in the last weeks), and if something changed in fiona / gdal / the conda packages to make this problem worse / easier to run into. (unless this is just coincidental)

fargiolas commented 3 years ago

Now, what I am less sure about is why it seems that there now new issues with this (given that multiple people posted here in the last weeks), and if something changed in fiona / gdal / the conda packages to make this problem worse / easier to run into. (unless this is just coincidental)

I've been trying to track this on my setup (macos, arm64, packages installed with brew and pip3, no conda) with little success. The first fiona import geopandas/io/file.py fails with:

>>> import geopandas Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/homebrew/lib/python3.9/site-packages/geopandas/__init__.py", line 7, in <module> from geopandas.io.file import _read_file as read_file # noqa File "/opt/homebrew/lib/python3.9/site-packages/geopandas/io/file.py", line 20, in <module> from fiona import Env as fiona_env File "/opt/homebrew/lib/python3.9/site-packages/fiona/__init__.py", line 85, in <module> with fiona._loading.add_gdal_dll_directories(): AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)

But if I wrap a try/except around all the fiona imports in fiona/__init__.py after with fiona._loading.add_gdal_dll_directories(): the exception I get is the following:

>>> import geopandas exception: dlopen(/opt/homebrew/lib/python3.9/site-packages/fiona/ogrext.cpython-39-darwin.so, 2): Symbol not found: _GEOSArea Referenced from: /opt/homebrew/opt/libspatialite/lib/libspatialite.7.dylib Expected in: flat namespace in /opt/homebrew/opt/libspatialite/lib/libspatialite.7.dylib Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/homebrew/lib/python3.9/site-packages/geopandas/__init__.py", line 7, in <module> from geopandas.io.file import _read_file as read_file # noqa File "/opt/homebrew/lib/python3.9/site-packages/geopandas/io/file.py", line 12, in <module> import fiona File "/opt/homebrew/lib/python3.9/site-packages/fiona/__init__.py", line 112, in <module> __gdal_version__ = get_gdal_release_name() NameError: name 'get_gdal_release_name' is not defined

The same doesn't happen if I import fiona directly, only from geopandas. I tried setting gdal datadir with export GDAL_DATA=$(gdal-config --datadir) but it does not help.

There are some old mac specific bugs about the missing symbol in libspatialite, see e.g. https://github.com/Homebrew/homebrew-core/issues/5161 maybe some recent change triggered them again.

Still digging, I'll comment if I find anything new, if you have some test you'd like me to run let me know.

907Resident commented 3 years ago

Hi, on MacOS using pip installs I also had this error when importing geopandas. Python 3.9.2.

I changed pyproj from 3.0.1 to 3.0.0 and now the error goes away as long as I import fiona before geopandas.

If I only import geopandas without fiona first, I still get the "Circular import" error message that others reported.

This worked for me using the same Python interpreter but on a Windows. However, I can only make it work using a script in vscode and not in notebook

rbuffat commented 3 years ago

I just got the AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import) with geopandas on Linux having a broken local install of Fiona.

The way geopandas imports Fiona (or how Fiona uses the _loading module) seems to suppress error messages when importing Fiona.

Steps to reproduce: Install fiona using pip. Remove /usr/lib/python3.9/site-packages/Fiona.libs/libgdal-d962753e.so.28.0.2

Python 3.9.4 (default, Apr 20 2021, 15:51:38) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fiona
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/site-packages/fiona/__init__.py", line 86, in <module>
    from fiona.collection import BytesCollection, Collection
  File "/usr/lib/python3.9/site-packages/fiona/collection.py", line 11, in <module>
    from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
ImportError: libgdal-d962753e.so.28.0.2: cannot open shared object file: No such file or directory
>>> import geopandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/site-packages/geopandas/__init__.py", line 7, in <module>
    from geopandas.io.file import _read_file as read_file  # noqa
  File "/usr/lib/python3.9/site-packages/geopandas/io/file.py", line 12, in <module>
    import fiona
  File "/usr/lib/python3.9/site-packages/fiona/__init__.py", line 85, in <module>
    with fiona._loading.add_gdal_dll_directories():
AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)
amylastuka commented 3 years ago

Probably a different bug, but I'm seeing a similar circular import error when I import geopandas (fiona 1.18.18, geopandas 0.9.0, gdal 3.2.2, installed with pip and brew on macos). The error goes away if I import fiona before geopandas.

Thank you, I had this same problem and importing fiona first fixed it for me!

BlaneG commented 3 years ago

The solution that worked for me: Delete all of the directories in '..\AppData\Roaming\jupyter\kernels'. If this is not where your kernel specs are located you can find yours with jupyter kernelspec list from within the environment that is causing the error.

As @jorisvandenbossche pointed out, it seems that muddeled jupyter environments were causing this issue for me. I could generate the error message in a jupyter notebook with import fiona, but I had no issue when executing import fiona directly in an ipython kernel launched from Anaconda Prompt with ipython. The stack trace from my jupyter notebook showed that fiona was being loaded from a conda environment that was different from the environment I was launching jupyter lab from.

Before figuring out the above solution I had tried the following workarounds mentioned above with no success:

  1. import fiona before geopandas recommended by @fargiolas
  2. launch notebook after activating conda environment recommended by @jorisvandenbossche
  3. install nb_conda_kernels recommended by @jorisvandenbossche
  4. using sys.path.append('{path-to-.conda}\envs\env\Library\bin') recommended by @jorisvandenbossche
MLArchaeology commented 3 years ago

OS: Windows 10 Setup: PyCharm CE 2019. Python 3.8 Conda Env _fiona 1.8.18 py39h9f1b0431 conda-forge

Using pyinstaller I got the "partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)" error.

Implemented import fiona before geopandas recommended by @fargiolas

100% Error Gone! BAM! Different Error to solve!

jorisvandenbossche commented 3 years ago

I have a question for those of you who commented here and where "import fiona before geopandas" fixes the issue.

To be able to investigate this issue, I would like to reproduce it (I have access to a Linux and Windows machine), but I currently don't know how.

If some people that have this specific issue could describe their environment / how they installed geopandas and fiona / etc, that would be very helpful!

fargiolas commented 3 years ago

@jorisvandenbossche I'm one of those but I only reproduced it on a macbook m1 (arm64) with packages installed from brew and pip, not sure if I can help. It's been a while since I looked into this, I should be able to run a quick test this evening to see if the issue is still there.

jorisvandenbossche commented 3 years ago

@fargiolas if you would be able to check if you can still reproduce the issue, that would certainly be welcome! (let me know if you succeed, at which point I have a bunch of questions / things to test :) Are you using gitter? that might be a bit easier to chat about it)

nmarchio commented 2 years ago

Not sure if this is useful but I encountered this exact circular import error today and was able to resolve it by pinning my pygeos version to 0.10.2 instead of the most recent 0.12.0. I'm running it on a Macbook M1 (arm64) architecture. Note, this conda environment installs Fiona version 1.8.20.

conda create --name pygeospatial python=3.9.7 --yes
source activate pygeospatial 
conda install -c conda-forge pygeos=0.10.2 --yes
conda install -c conda-forge geopandas=0.10.2 --yes 
KevinChen9412 commented 2 years ago

One workaround is to install notebook in the environment you want to use, first activate the environment, and then start the notebook from that environment

I am having this Circular imports issue only when i try to import fiona in jupyter notebook (everything works fine in my Pycharm). This workaround fixed my issue. @jorisvandenbossche

adamjstewart commented 2 years ago

We just hit the same circular import issue in our CI today, should this issue be reopened?

theroggy commented 2 years ago

A few months ago I had the issue on my CI tests on MacOS, but it disappeared again a few weeks later, and since yesterday the problem popped up on the windows tests...

adamjstewart commented 2 years ago

I think it's related to this new version of the conda feedstock released yesterday: https://github.com/conda-forge/fiona-feedstock/pull/191

adamjstewart commented 2 years ago

If only we had Windows wheels and didn't need to rely on conda...

MattBrst commented 2 years ago

I had the same error messages as mentioned above on a fresh Python 3.10 install from Anaconda on Windows 10:

import fiona

resulting in the following error message: ImportError: DLL load failed while importing ogrext: Kan opgegeven procedure niet vinden.

import geopandas
import fiona

resulting in another error message: AttributeError: partially initialized module 'fiona' has no attribute 'loading' (most likely due to a circular import).

The issue was solved by the following order of importing packages:

from osgeo import gdal
import fiona
import geopandas
mlnrt commented 2 years ago

I also ran into that same issue and it drove me crazy for days. In my code, in one of my own library I have the following imports:

import pandas as pd
import pygeos
import geopandas as gpd
from fiona.errors import DriverError

This resulted in the below behaviors:

The packages were installed as follows using conda:

conda install -c conda-forge -y fiona==1.8.21 geopandas==0.10.2 pygeos==0.12.0

I tried installing GDAL from the GISInternals website as recommended on the Fiona PiPy page. But it did not solve the issue. The Fiona installation instruction also mention "You will also need to download the compiled libraries and headers (include files)." But there is no instructions as to where to put those downloaded files. I tried to install Fiona from wheel file which can be obtained from Christoph Gohlke’s website. But that did not make a difference compared to the conda-forge installation.

Also in PyCharm/DataSpell/VSCode/Jupyter, the conda environment seems to be activated correctly. The os.environ["PATH"] and os.environ["GDAL_DATA"] are there and correct.

None of the mentioned solutions mentioned in the thread above (or other forums) worked:

In the end what worked for me was the following:

  1. Uninstall the direct Python 3.9 and 3.10 installations I had on top of Anaconda 3
  2. Uninstall Anaconda 3
  3. Reinstall Anaconda 3 with the options "Add Anaconda to my PATH environment variable" and "Register Anaconda as my default Python 3.9"
  4. Recreate my conda environment

I also noticed the following issue: If I import tensorflow before fiona.errors and geopandas then I get into the same error again "AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)"

jorisvandenbossche commented 2 years ago

@mlnrt I am sorry to hear the troubles it caused, but thanks for the detailed write-up (it is a really annoying issue ..).
In the unlikely case you would still be up for some more testing (now that you finally have it working ..), I would be interested to know if you run into the same issue if installing pyogrio instead of fiona (if it works there, that might give us some pointers if we can change something about fiona)

mlnrt commented 2 years ago

@jorisvandenbossche Thank you for reach out. I tried to create a new conda environment with pyogrio but it only installs pygeos. geopandas is a requirement but it doesn't installs it by default so I had to add it to the list of package and geopandas will install fiona. So by installing pyogrio and geopandas , compare to before I just end up with pyogrio on top of fiona, geopandas and pygeos. So I am not sure why it would make a difference. I mean it worked, but since I completely rebuilt my Windows Anaconda and Python environment it also works without pyogrio. The other problem is that to really test, I would need to go back to my previous Anaconda + Python 3.9 + Python 3.10 environment which I don't have anymore. In my current windows environment Fiona, Geopandas and PyGeos do work correctly.

jorisvandenbossche commented 2 years ago

compare to before I just end up with pyogrio on top of fiona, geopandas and pygeos. So I am not sure why it would make a difference.

It could still be the case that importing fiona fails, and importing pyogrio works. But of course that's only useful to test in the env where fiona was actually failing to be imported. And if you don't have that installation at hand anymore, don't worry about it (not need to go back to your previous not-working setup!)

gmerritt123 commented 1 year ago

I had the same error messages as mentioned above on a fresh Python 3.10 install from Anaconda on Windows 10:

import fiona

resulting in the following error message: ImportError: DLL load failed while importing ogrext: Kan opgegeven procedure niet vinden.

import geopandas
import fiona

resulting in another error message: AttributeError: partially initialized module 'fiona' has no attribute 'loading' (most likely due to a circular import).

The issue was solved by the following order of importing packages:

from osgeo import gdal
import fiona
import geopandas

+1 this worked for me. Thanks so much. On a python 3.8 conda forge environment built up a few weeks ago.

If this resolves the issue, the issue has to be that the gdal path is messed up eh? Hard importing gdal in gives fiona/geopd the right path to point to but without it it fails?

This is the second conda environment on this machine that needs gdal/fiona/geopandas (porting stuff from 3.7 to 3.8)... but there's only one path/one specific path each fiona/geopd can point to for all of them somehow? Any advice on updating that path (where and to what) would likely perma-solve the issue....