search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.29k stars 916 forks source link

[Feature] Support the metadata management #1155

Closed xwmr-max closed 1 year ago

xwmr-max commented 1 year ago

Search before asking

Motivation

Support the metadata management, such as the expire snapshots,the rewrite manifests,and so on.

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

JingsongLi commented 1 year ago

For Flink, maybe the only way is adding Actions. For Spark, we can add procedures after https://github.com/apache/incubator-paimon/pull/581

xwmr-max commented 1 year ago

I will write actions in flink first, and as for spark, I will add related abilities after #581 @SteNicholas .

JingsongLi commented 1 year ago

I will write actions in flink first, and as for spark, I will add related abilities after #581 @SteNicholas .

Thanks @xwmr-max ! Feel free to open PRs.