search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.35k stars 928 forks source link

[Feature] Paimon Rust #3674

Open Xuanwo opened 3 months ago

Xuanwo commented 3 months ago

Search before asking

Motivation

It's extremely difficult for non-JVM languages like Rust and Python to interact with Paimon tables.

If we implement Paimon in Rust, it will enable a lot of use cases.

For instance, we could facilitate reading and writing Paimon data within Databend (in Rust) and DuckDB (in C++). Additionally, by creating paimon-py based on paimon-rust, we can make Paimon accessible in Python environments.

Solution

Create a pure rust implementation of the paimon spec similar to Iceberg project: iceberg-rust.

Anything else?

I'm willing to help make it happen. We can start by creating a paimon-rust repo.

Are you willing to submit a PR?

Xuanwo commented 3 months ago

The work has been started at https://github.com/apache/paimon-rust, welcome to join in!

Xuanwo commented 3 months ago

The tracking issues of paimon-rust 0.1 release, welcome to check out: https://github.com/apache/paimon-rust/issues/3