Open weiji14 opened 1 month ago
Thanks for the report! This is a tricky one, and appears to be an order of operations, possibly loading issue loading the GDAL library multiple times from different packages (e.g., pyogrio, rasterio).
I can reproduce locally by installing a Conda env based on your environment.
I'm seeing that it appears to work properly at first and then fail later. To test this, I edited the geopandas/io/file.py::__read_file_pyogrio
to print drivers after importing pyogrio.
When I run the equivalent of your make html
(with logging): PYGMT_USE_EXTERNAL_DISPLAY="false" sphinx-build -b html -d _build/doctrees -j auto . _build/html -v
I see that it first successfully loads all drivers when running examples/gallery/maps/choropleth_map.py
. It then runs examples/gallery/maps/tilemaps.py
and I see warnings about rio
, which suggests that it is loading rasterio
somewhere in the stack at that point. After that, it runs examples/gallery/lines/roads.py
and now lists no drivers before crashing.
If I run the failing file directly:
PYGMT_USE_EXTERNAL_DISPLAY="false" python examples/gallery/lines/roads.py
It loads all drivers successfully and works as expected.
If I comment out the contents of examples/gallery/maps/tilemaps.py
and then run sphinx-build again (above command), then roads.py
is successful. So - something in tilemaps.py
is causing GDAL loading to get messed up.
I've isolated the problematic line that changes the behavior of loading drivers in pyogrio. If we import pyogrio and list drivers before this, they are as expected. If we import pyogrio after this line, no drivers are loaded.
I don't use xarray, so I'm not able to go much further down this rabbit hole. Based on what I'm seeing, this isn't an error that is caused directly by pyogrio (though please follow up here if you discover otherwise). It seems to be based on multiple libraries loading GDAL differently and then based on order of operations, seems to be leaving the drivers available in GDAL in a bad state when subsequent packages try to load GDAL.
@snowman2 do you have any ideas about what might be causing this order of operations driver loading issue based on rioxarray
? It seems that loading GDAL drivers works fine in pyogrio before a call to rioxarray
, and then does not work properly afterward. That is, if rioxarray is called before pyogrio, we don't get drivers from GDAL, but if it is called after, then things work properly.
rioxarray imports rasterio. Did you try importing rasterio before/after pyogrio?
Here is a much smaller working example (compared to building the PyGMT documentation) to reproduce the issue:
import geopandas as gpd
gdf = gpd.read_file("https://www2.census.gov/geo/tiger/TIGER2015/PRISECROADS/tl_2015_15_prisecroads.zip")
import pygmt
fig = pygmt.Figure()
fig.tilemap(region=[-157.84, -157.8, 21.255, 21.285], projection="M12c", zoom=14, frame="afg")
fig.show()
gdf = gpd.read_file("https://www2.census.gov/geo/tiger/TIGER2015/PRISECROADS/tl_2015_15_prisecroads.zip")
Importing rasterio first works fine (as does simply importing rioxarray
).
Thanks for the smaller example @seisman .
It looks like an offending section in pygmt/src/tilemap.py
is this one (if I comment it out, your example works as expected):
with Session() as lib:
with lib.virtualfile_in(check_kind="raster", data=raster) as vingrd:
lib.call_module(
module="grdimage", args=build_arg_list(kwargs, infile=vingrd)
)
Thanks so much @brendan-ward for taking the time to debug this! Your comment at https://github.com/geopandas/pyogrio/issues/448#issuecomment-2219139470 makes a lot of sense (to me at least), I spent hours debugging this and did notice how the first choropleth_map.py
example worked fine, but roads.py
didn't, but didn't realize how tilemaps.py
in the middle was possibly causing issues.
@seisman, it seems like GMT might also be calling GDALAllRegister
(e.g. at https://github.com/GenericMappingTools/gmt/blob/804d7674b890fe0e83411b1083ddbb2128b49847/src/gmt_gdalread.c#L721), would the interaction between pyogrio/GMT/rasterio(GDAL) be causing issues?
it seems like GMT might also be calling
GDALAllRegister
(e.g. at https://github.com/GenericMappingTools/gmt/blob/804d7674b890fe0e83411b1083ddbb2128b49847/src/gmt_gdalread.c#L721), would the interaction between pyogrio/GMT/rasterio(GDAL) be causing issues?
It's likely. Here is an even smaller example to reproduce it, without calling the Figure.tilemap
/load_tile_map
method/function. So it means it's not directly related to rioxarray
/rasterio
. To me, it seems as long as GMT tries to read an image, it crashes:
import geopandas as gpd
gdf = gpd.read_file("https://www2.census.gov/geo/tiger/TIGER2015/PRISECROADS/tl_2015_15_prisecroads.zip")
import pygmt
fig = pygmt.Figure()
fig.grdimage("@earth_day_01d")
fig.show()
gdf = gpd.read_file("https://www2.census.gov/geo/tiger/TIGER2015/PRISECROADS/tl_2015_15_prisecroads.zip")
The upstream documentation says (https://gdal.org/api/raster_c_api.html#_CPPv415GDALAllRegisterv):
This function should generally be called once at the beginning of the application.
but apparently, it's called multiple times even in GMT.
We've been hitting into an issue on pyogrio not being able to detect GDAL drivers in our Sphinx docs build CI since two weeks ago (https://github.com/GenericMappingTools/pygmt/issues/3301), with errors like:
pyogrio.errors.DataSourceError: No driver registered.
pyogrio.errors.DataSourceError: Could not obtain driver: OGR_GMT (check that it was installed correctly into GDAL)
We're using
geopandas=1.0.1 (pyhd8ed1ab_0)
andpyogrio=0.9.0 (py312h8ad7a51_0)
installed from conda-forge, which includes the OGR_GMT driver (installed inGDAL=3.9.0 (py312h86af8fa_5)
). This issue has been very hard to reproduce, because things work when we test things directly, but fail when running the scripts withsphinx-build
. E.g. this code example:would consistently produce a
pyogrio.errors.DataSourceError: No driver registered
error when ran as part ofsphinx-build
:or produce the expected output when ran directly in a Python script or Jupyter notebook:
My guess is that the GDAL drivers are not being registered properly somehow. This was supposedly fixed in https://github.com/geopandas/pyogrio/pull/145 (see also https://github.com/geopandas/pyogrio/issues/144), but there might be certain cases where the loading doesn't happen correctly? We have a workaround right now that forces the driver load like so:
but given that
_register_drivers
is a private method, we would prefer not to rely on it. We're opening this issue to try to figure out where the GDAL driver loading logic might be failing even after #145. Unsure if puttingGDALAllRegister()
back in _io.pyx and/or _ogr.pyx would help, or if there is another solution we can try.