search

ibis-project / ibis

the portable Python dataframe library
https://ibis-project.org
Apache License 2.0
4.51k stars 549 forks source link

feat: Apache Sedona backend #7999

Open lostmygithubaccount opened 5 months ago

lostmygithubaccount commented 5 months ago

Is your feature request related to a problem?

https://sedona.apache.org

Apache Sedona™ is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark and Apache Flink, with a set of out-of-the-box distributed Spatial Datasets and Spatial SQL that efficiently load, process, and analyze large-scale spatial data across machines.

Apache Sedona has a Python interface (https://pypi.org/project/apache-sedona/) but could benefit from the usability of Ibis with its geospatial support. Apache Sedona supports Apache Flink and Apache Spark streaming. Ibis is in the process of releasing an Apache Flink backend.

This would be an interesting option for distributed geospatial work.

Describe the solution you'd like

Implement an Apache Sedona backend for Ibis streaming + geospatial operations.

What version of ibis are you running?

main

What backend(s) are you using, if any?

n/a

Code of Conduct

jiayuasu commented 5 months ago

@lostmygithubaccount Created a corresponding issue on Sedona Github repo and hopefully someone will be willing to take a lead!

lostmygithubaccount commented 5 months ago

@jiayuasu thanks! depending on the timing of this, we're currently undergoing a large internal refactor moving to a new relational operations model and swapping out SQLAlchemy for SQLGlot. I mention this because depending on the timing of when this would be added, we may want to start with the new system directly

once somebody is ready to take it up let us know here on GH and we can hop on a call or discuss further in our Zulip chat (or GH)

we are also going to announce streaming backend support in Ibis v8, targeting the end of January. I assume this won't make it in but could be worth mentioning depending on the state of things