search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.1k stars 834 forks source link

[Feature] Add a listener mechanism for Paimon #1811

Open shidayang opened 10 months ago

shidayang commented 10 months ago

Search before asking

Motivation

In the process of integrating services like the Amoro Lakehouse management system with Paimon, it is necessary to timely perceive the state changes of Paimon's tables, such as whether there is data commit, the current compaction status of the write task, and so on. Therefore, a Listener mechanism is introduced to synchronize the state to the outside world.

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

shidayang commented 10 months ago

Design document: https://cwiki.apache.org/confluence/display/PAIMON/PIP-8%3A+Introduce+listeners+for+Paimon

hililiwei commented 7 months ago

hi, @shidayang we are trying to use Paimon recently and have made some enhancements, including pub/sub. I raised a PR, could you please help review it?