Closed osjahputera closed 4 months ago
@jorisvandenbossche - should this be updated to handle the absence of crs
in column metadata? https://github.com/geopandas/dask-geopandas/blob/3489a1cbafbeda3c0d4493133112969268e58d66/dask_geopandas/io/arrow.py#L36
Apologies for the slow reply here. Thanks @osjahputera for the report! And @tschaub indeed we should not just assume the 'crs" key is present, thanks for the catch. Putting up a quick fix for this.
I am using a small geoparquet file below (33 MB ~ building footprints in Singapore (SGP)), but the intention is to process larger geoparquet files covering much larger countries.
https://data.source.coop/vida/google-microsoft-open-buildings/geoparquet/by_country/country_iso=SGP/SGP.parquet
I am getting this error when reading the
SGP.parquet
file using the following code:Error messages:
``` --------------------------------------------------------------------------- KeyError Traceback (most recent call last) File ~\anaconda3\envs\geo_env\Lib\site-packages\dask\backends.py:141, in CreationDispatch.register_inplace.The original error message contains the string
crs
. Is there a problem in getting the coordinate reference system when reading geoparquet file fromdask_geopandas
?The same
SGP.parquet
file can be read correctly usinggeopandas
as follows:The
conda
environment is built as follows:I initially built the conda env following the instruction provided in the README.md
However, this does not include
pyarrow.parquet
, thus the functiongeopandas.read_parquet()
is not available.The list of installed conda packages and versions.
# Name Version Build Channel affine 2.4.0 pyhd8ed1ab_0 conda-forge anyio 4.3.0 pyhd8ed1ab_0 conda-forge argon2-cffi 23.1.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py311ha68e1ae_4 conda-forge arrow 1.3.0 pyhd8ed1ab_0 conda-forge asttokens 2.4.1 pyhd8ed1ab_0 conda-forge async-lru 2.0.4 pyhd8ed1ab_0 conda-forge attrs 23.2.0 pyh71513ae_0 conda-forge aws-c-auth 0.7.16 h0a3593b_5 conda-forge aws-c-cal 0.6.10 hd481e46_1 conda-forge aws-c-common 0.9.13 hcfcfb64_0 conda-forge aws-c-compression 0.2.18 hd481e46_1 conda-forge aws-c-event-stream 0.4.2 h9555df4_3 conda-forge aws-c-http 0.8.1 h0b45682_4 conda-forge aws-c-io 0.14.4 h08270f9_2 conda-forge aws-c-mqtt 0.10.2 hababdb7_3 conda-forge aws-c-s3 0.5.1 h228a0d0_6 conda-forge aws-c-sdkutils 0.1.15 hd481e46_1 conda-forge aws-checksums 0.1.18 hd481e46_1 conda-forge aws-crt-cpp 0.26.2 hb1302ff_5 conda-forge aws-sdk-cpp 1.11.267 h93f5800_1 conda-forge babel 2.14.0 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.12.3 pyha770c72_0 conda-forge bleach 6.1.0 pyhd8ed1ab_0 conda-forge blosc 1.21.5 hdccc3a2_0 conda-forge bokeh 3.3.4 pyhd8ed1ab_0 conda-forge boost-cpp 1.78.0 h9f4b32c_4 conda-forge brotli 1.1.0 hcfcfb64_1 conda-forge brotli-bin 1.1.0 hcfcfb64_1 conda-forge brotli-python 1.1.0 py311h12c1d0e_1 conda-forge bzip2 1.0.8 he774522_0 c-ares 1.26.0 hcfcfb64_0 conda-forge ca-certificates 2024.2.2 h56e8100_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cairo 1.16.0 hdecc03f_1016 conda-forge certifi 2024.2.2 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py311ha68e1ae_0 conda-forge cfitsio 4.2.0 h9ebe7e4_0 conda-forge charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge click 8.1.7 win_pyh7428d3b_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.7.2 pyhd8ed1ab_1 conda-forge cloudpickle 3.0.0 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge comm 0.2.1 pyhd8ed1ab_0 conda-forge contextily 1.5.0 pyhd8ed1ab_0 conda-forge contourpy 1.2.0 py311h005e61a_0 conda-forge curl 8.5.0 hd5e4a3a_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge cytoolz 0.12.3 py311ha68e1ae_0 conda-forge dask 2024.2.0 pyhd8ed1ab_0 conda-forge dask-core 2024.2.0 pyhd8ed1ab_0 conda-forge dask-geopandas 0.3.1 pyhd8ed1ab_1 conda-forge debugpy 1.8.1 py311h12c1d0e_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge distributed 2024.2.0 pyhd8ed1ab_0 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge exceptiongroup 1.2.0 pyhd8ed1ab_2 conda-forge executing 2.0.1 pyhd8ed1ab_0 conda-forge expat 2.5.0 h63175ca_1 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_1 conda-forge fontconfig 2.14.2 hbde0cde_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.49.0 py311ha68e1ae_0 conda-forge fqdn 1.5.1 pyhd8ed1ab_0 conda-forge freetype 2.12.1 hdaf720e_2 conda-forge freexl 1.0.6 h67ca5e6_1 conda-forge fsspec 2024.2.0 pyhca7485f_0 conda-forge gdal 3.7.0 py311h68c9619_3 conda-forge geographiclib 1.52 pyhd8ed1ab_0 conda-forge geopandas-base 0.14.3 pyha770c72_0 conda-forge geopy 2.4.1 pyhd8ed1ab_0 conda-forge geos 3.11.2 h1537add_0 conda-forge geotiff 1.7.1 h4e61e90_11 conda-forge gettext 0.21.1 h5728263_0 conda-forge h11 0.14.0 pyhd8ed1ab_0 conda-forge h2 4.1.0 pyhd8ed1ab_0 conda-forge hdf4 4.2.15 h1334946_6 conda-forge hdf5 1.14.1 nompi_h73e8ff5_100 conda-forge hpack 4.0.0 pyh9f0ad1d_0 conda-forge httpcore 1.0.4 pyhd8ed1ab_0 conda-forge httpx 0.27.0 pyhd8ed1ab_0 conda-forge hyperframe 6.0.1 pyhd8ed1ab_0 conda-forge icu 72.1 h63175ca_0 conda-forge idna 3.6 pyhd8ed1ab_0 conda-forge importlib-metadata 7.0.1 pyha770c72_0 conda-forge importlib_metadata 7.0.1 hd8ed1ab_0 conda-forge importlib_resources 6.1.1 pyhd8ed1ab_0 conda-forge intel-openmp 2024.0.0 h57928b3_49841 conda-forge ipykernel 6.29.2 pyha63f2e9_0 conda-forge ipython 8.21.0 pyh7428d3b_0 conda-forge isoduration 20.11.0 pyhd8ed1ab_0 conda-forge jedi 0.19.1 pyhd8ed1ab_0 conda-forge jinja2 3.1.3 pyhd8ed1ab_0 conda-forge joblib 1.3.2 pyhd8ed1ab_0 conda-forge json5 0.9.17 pyhd8ed1ab_0 conda-forge jsonpointer 2.4 py311h1ea47a8_3 conda-forge jsonschema 4.21.1 pyhd8ed1ab_0 conda-forge jsonschema-specifications 2023.12.1 pyhd8ed1ab_0 conda-forge jsonschema-with-format-nongpl 4.21.1 pyhd8ed1ab_0 conda-forge jupyter-lsp 2.2.2 pyhd8ed1ab_0 conda-forge jupyter_client 8.6.0 pyhd8ed1ab_0 conda-forge jupyter_core 5.7.1 py311h1ea47a8_0 conda-forge jupyter_events 0.9.0 pyhd8ed1ab_0 conda-forge jupyter_server 2.12.5 pyhd8ed1ab_0 conda-forge jupyter_server_terminals 0.5.2 pyhd8ed1ab_0 conda-forge jupyterlab 4.1.2 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.3.0 pyhd8ed1ab_1 conda-forge jupyterlab_server 2.25.3 pyhd8ed1ab_0 conda-forge kealib 1.5.1 h96def13_4 conda-forge kiwisolver 1.4.5 py311h005e61a_1 conda-forge krb5 1.21.2 heb0366b_0 conda-forge lcms2 2.15 h3e3b177_1 conda-forge lerc 4.0.0 h63175ca_0 conda-forge libabseil 20230802.1 cxx17_h63175ca_0 conda-forge libaec 1.1.2 h63175ca_1 conda-forge libarchive 3.6.2 h6f8411a_1 conda-forge libarrow 15.0.0 h04ba218_5_cpu conda-forge libarrow-acero 15.0.0 h63175ca_5_cpu conda-forge libarrow-dataset 15.0.0 h63175ca_5_cpu conda-forge libarrow-flight 15.0.0 he112ba8_5_cpu conda-forge libarrow-flight-sql 15.0.0 h8f0bfdc_5_cpu conda-forge libarrow-gandiva 15.0.0 h41a35cd_5_cpu conda-forge libarrow-substrait 15.0.0 h7aa34db_5_cpu conda-forge libblas 3.9.0 21_win64_mkl conda-forge libbrotlicommon 1.1.0 hcfcfb64_1 conda-forge libbrotlidec 1.1.0 hcfcfb64_1 conda-forge libbrotlienc 1.1.0 hcfcfb64_1 conda-forge libcblas 3.9.0 21_win64_mkl conda-forge libcrc32c 1.1.2 h0e60522_0 conda-forge libcurl 8.5.0 hd5e4a3a_0 conda-forge libdeflate 1.18 hcfcfb64_0 conda-forge libevent 2.1.12 h3671451_1 conda-forge libexpat 2.5.0 h63175ca_1 conda-forge libffi 3.4.4 hd77b12b_0 libgdal 3.7.0 h123a834_3 conda-forge libglib 2.78.1 he8f3873_0 conda-forge libgoogle-cloud 2.17.0 h7ece6b4_2 conda-forge libgoogle-cloud-storage 2.17.0 hb581fae_2 conda-forge libgrpc 1.60.1 h0bf0bfa_0 conda-forge libhwloc 2.9.3 default_haede6df_1009 conda-forge libiconv 1.17 hcfcfb64_2 conda-forge libjpeg-turbo 2.1.5.1 hcfcfb64_1 conda-forge libkml 1.3.0 hd45a9bc_1016 conda-forge liblapack 3.9.0 21_win64_mkl conda-forge libnetcdf 4.9.2 nompi_h5902ca5_107 conda-forge libparquet 15.0.0 h7ec3a38_5_cpu conda-forge libpng 1.6.42 h19919ed_0 conda-forge libpq 15.6 h94c9ec1_0 conda-forge libprotobuf 4.25.1 hb8276f3_2 conda-forge libre2-11 2023.06.02 h8c5ae5e_0 conda-forge librttopo 1.1.0 he1da8c1_13 conda-forge libsodium 1.0.18 h8d14728_1 conda-forge libspatialite 5.0.1 hf8d749d_27 conda-forge libsqlite 3.45.1 hcfcfb64_0 conda-forge libssh2 1.11.0 h7dfc565_0 conda-forge libthrift 0.19.0 ha2b3283_1 conda-forge libtiff 4.5.1 h6c8260b_1 conda-forge libutf8proc 2.8.0 h82a8f57_0 conda-forge libwebp-base 1.3.2 hcfcfb64_0 conda-forge libxcb 1.15 hcd874cb_0 conda-forge libxml2 2.12.5 hc3477c8_0 conda-forge libzip 1.10.1 h1d365fa_3 conda-forge libzlib 1.2.13 hcfcfb64_5 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lz4 4.3.3 py311haddf500_0 conda-forge lz4-c 1.9.4 hcfcfb64_0 conda-forge lzo 2.10 he774522_1000 conda-forge m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge markupsafe 2.1.5 py311ha68e1ae_0 conda-forge matplotlib-base 3.8.3 py311h6e989c2_0 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mercantile 1.2.1 pyhd8ed1ab_0 conda-forge mistune 3.0.2 pyhd8ed1ab_0 conda-forge mkl 2024.0.0 h66d3029_49657 conda-forge msgpack-python 1.0.7 py311h005e61a_0 conda-forge msys2-conda-epoch 20160418 1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge nb_conda_kernels 2.3.1 pyh7428d3b_4 conda-forge nbclient 0.8.0 pyhd8ed1ab_0 conda-forge nbconvert-core 7.16.1 pyhd8ed1ab_0 conda-forge nbformat 5.9.2 pyhd8ed1ab_0 conda-forge nest-asyncio 1.6.0 pyhd8ed1ab_0 conda-forge notebook 7.1.0 pyhd8ed1ab_0 conda-forge notebook-shim 0.2.4 pyhd8ed1ab_0 conda-forge numpy 1.26.4 py311h0b4df5a_0 conda-forge openjpeg 2.5.0 ha2aaf27_2 conda-forge openssl 3.2.1 hcfcfb64_0 conda-forge orc 1.9.2 hf6f83f4_1 conda-forge overrides 7.7.0 pyhd8ed1ab_0 conda-forge packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 2.2.0 py311hf63dbb6_0 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge partd 1.4.1 pyhd8ed1ab_0 conda-forge pcre2 10.40 h17e33f8_0 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 10.0.0 py311hde623f7_0 conda-forge pip 23.3.1 py311haa95532_0 pixman 0.43.2 h63175ca_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge platformdirs 4.2.0 pyhd8ed1ab_0 conda-forge poppler 23.05.0 h45d20d0_1 conda-forge poppler-data 0.4.12 hd8ed1ab_0 conda-forge postgresql 15.6 h94c9ec1_0 conda-forge proj 9.2.1 h660b3b0_0 conda-forge prometheus_client 0.20.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.42 pyha770c72_0 conda-forge psutil 5.9.8 py311ha68e1ae_0 conda-forge pthread-stubs 0.4 hcd874cb_1001 conda-forge pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pyarrow 15.0.0 py311h6a6099b_5_cpu conda-forge pyarrow-hotfix 0.6 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygeos 0.14 py311h343093d_1 conda-forge pygments 2.17.2 pyhd8ed1ab_0 conda-forge pyogrio 0.6.0 py311h03c997e_1 conda-forge pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge pyproj 3.6.1 py311hcff2a09_0 conda-forge pysocks 1.7.1 pyh0701188_6 conda-forge python 3.11.5 he1021f5_0 python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.19.1 pyhd8ed1ab_0 conda-forge python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge python-tzdata 2024.1 pyhd8ed1ab_0 conda-forge python_abi 3.11 2_cp311 conda-forge pytz 2024.1 pyhd8ed1ab_0 conda-forge pywin32 306 py311h12c1d0e_2 conda-forge pywinpty 2.0.12 py311h12c1d0e_0 conda-forge pyyaml 6.0.1 py311ha68e1ae_1 conda-forge pyzmq 25.1.2 py311h9250fbb_0 conda-forge rasterio 1.3.8 py311h58bba51_0 conda-forge re2 2023.06.02 hcbb65ff_0 conda-forge referencing 0.33.0 pyhd8ed1ab_0 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge rpds-py 0.18.0 py311hc37eb10_0 conda-forge send2trash 1.8.2 pyh08f2357_0 conda-forge setuptools 68.2.2 py311haa95532_0 shapely 2.0.1 py311h343093d_1 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.10 hfb803bf_0 conda-forge sniffio 1.3.0 pyhd8ed1ab_0 conda-forge snuggs 1.4.7 py_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.5 pyhd8ed1ab_1 conda-forge sqlite 3.41.2 h2bbff1b_0 stack_data 0.6.2 pyhd8ed1ab_0 conda-forge tbb 2021.11.0 h91493d7_1 conda-forge tblib 3.0.0 pyhd8ed1ab_0 conda-forge terminado 0.18.0 pyh5737063_0 conda-forge tiledb 2.13.2 h3132609_0 conda-forge tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge tk 8.6.12 h2bbff1b_0 tomli 2.0.1 pyhd8ed1ab_0 conda-forge toolz 0.12.1 pyhd8ed1ab_0 conda-forge tornado 6.4 py311ha68e1ae_0 conda-forge traitlets 5.14.1 pyhd8ed1ab_0 conda-forge types-python-dateutil 2.8.19.20240106 pyhd8ed1ab_0 conda-forge typing-extensions 4.9.0 hd8ed1ab_0 conda-forge typing_extensions 4.9.0 pyha770c72_0 conda-forge typing_utils 0.1.0 pyhd8ed1ab_0 conda-forge tzdata 2023d h04d1e81_0 ucrt 10.0.22621.0 h57928b3_0 conda-forge uri-template 1.3.0 pyhd8ed1ab_0 conda-forge uriparser 0.9.7 h1537add_1 conda-forge urllib3 2.2.1 pyhd8ed1ab_0 conda-forge vc 14.2 h21ff451_1 vc14_runtime 14.38.33130 h82b7239_18 conda-forge vs2015_runtime 14.38.33130 hcb4865c_18 conda-forge wcwidth 0.2.13 pyhd8ed1ab_0 conda-forge webcolors 1.13 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 pyhd8ed1ab_2 conda-forge websocket-client 1.7.0 pyhd8ed1ab_0 conda-forge wheel 0.41.2 py311haa95532_0 win_inet_pton 1.1.0 pyhd8ed1ab_6 conda-forge winpty 0.4.3 4 conda-forge xerces-c 3.2.5 h63175ca_0 conda-forge xorg-libxau 1.0.11 hcd874cb_0 conda-forge xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge xyzservices 2023.10.1 pyhd8ed1ab_0 conda-forge xz 5.4.5 h8cc25b3_0 yaml 0.2.5 h8ffe710_2 conda-forge zeromq 4.3.5 h63175ca_0 conda-forge zict 3.0.0 pyhd8ed1ab_0 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 hcfcfb64_5 conda-forge zstd 1.5.5 h12be248_0 conda-forge