Open v2thegreat opened 9 months ago
I think this is working properly with the latest versions of these libraries:
dask 2024.2.1, pyarrow 15.0.0, geopandas 0.14.3, and dask-geopandas main. Can you try this script?
import geodatasets
import tempfile
import dask_geopandas
import geopandas
df = geopandas.read_file(geodatasets.get_url("geoda airbnb"))
ddf = dask_geopandas.from_geopandas(df, npartitions=2).spatial_shuffle()
ddf.to_parquet("/tmp/out.parquet")
dask_geopandas.read_parquet("/tmp/data.parquet").spatial_partitions
For me that outputs
0 POLYGON ((-87.59852 41.77113, -87.59852 42.023...
1 POLYGON ((-87.52414 41.64454, -87.52414 41.977...
dtype: geometry
Problem
When using Dask GeoPandas to write a GeoDataFrame to Parquet format, the spatial partitions appear not to be persisted correctly. This issue is observed when storing GeoPandas data with spatial information using Dask and the Parquet format.
Expected Behavior
The spatial partitions of the GeoDataFrame should be correctly persisted in the resulting Parquet files. This means that the spatial properties of the GeoDataFrame, such as geometry information, should be preserved during the conversion process, making it much faster to query.
Steps to Reproduce
Example Code
My end goal is to be able to query the data quickly and only grab the partitions that contain the bounds that I'm interested in, such as when using
clip
orcx