search

geoarrow / geoarrow-python

Python implementation of the GeoArrow specification
http://geoarrow.org/geoarrow-python/
Apache License 2.0
59 stars 3 forks source link

Read from clud storage using geoarrow.pyarrow.dataset #43

Closed nagyrobir closed 7 months ago

nagyrobir commented 9 months ago

Hi!

I was wondering if geoarrow-python has support for reading from geoparquet that is stored on a Azure blob storage? I know Pyarrow has it but uncertain if geoarrow-python has it. If it does have could you show an example of some sort?

Thank you !

paleolimbot commented 9 months ago

geoarrow-pyarrow is just a thin wrapper around pyarrow, so any call to pyarrow.dataset.dataset() should be supported by geoarrow.pyarrow.dataset.dataset() (you can see how the args are passed down here: https://github.com/geoarrow/geoarrow-python/blob/main/geoarrow-pyarrow/src/geoarrow/pyarrow/dataset.py#L499-L513 ). I don't have an example of how to open an Azure cloud file dataset off the top of my head and I'm not 100% certain that support has merged into pyarrow yet.

Note that geoarrow.pyarrow.dataset is pretty experimental and might change at any time 🙂 . I'm a little torn because the concept of geospatial-augmented dataset is very useful; however, part of the philosophy of geoarrow.pyarrow is that geometry should "just work" as a first-class data type (and special casing the dataset seems like a rather flagrant violation of this philosophy).

nagyrobir commented 7 months ago

Hi again!

Are there any plans to implement spatial predicates in the future? Like being able to join two geopyarrow tables based on inteserction of geometries or similar?

kylebarron commented 7 months ago

I can't speak for @paleolimbot of whether there will be a c-based spatial join in geoarrow-pyarrow, but I'm planning to have a spatial join on geoarrow tables in geoarrow-rust within the next couple releases. It'll interop seamlessly with geoarrow-pyarrow via #38

nagyrobir commented 7 months ago

I am a bit slow so type so be patient with me :), The code written in geoarrow-rust will be exposed/usable in geoarrow-pyarrow without any kind of extra coding effort?

paleolimbot commented 7 months ago

Anything Kyle exposes in geoarrow-rust will be accessible here eventually! But for the time being, there is no spatial join implementation in geoarrow-pyarrow (and it will be a while until there is a battle-tested version accessible here).

nagyrobir commented 7 months ago

Roger that! Thank you again for the work that you are doing!