apache / amoro

Apache Amoro (incubating) is a Lakehouse management system built on open data lake formats.
https://amoro.apache.org/
Apache License 2.0
863 stars 288 forks source link

[Feature]: Support switch from Changestore to Logstore automatically for flink source #264

Closed zhoujinsong closed 2 months ago

zhoujinsong commented 2 years ago

Description

Support switch from Changestore to Logstore automatically for flink source

Use case/motivation

Arctic use Changestore and Logstore to store CDC messages. Logstore use MQ(eg: Kafka) to distribute real-time cdc message. Changestore use iceberg table to store cdc message for analyse and provide near real-time incremental pull. In really use case, Logstore store message for several days and Changestore store message for a longer time. Flink source should support to read Changestore for early CDC messages and switch to Logstore when catch it up.

Describe the solution

Subtasks

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

github-actions[bot] commented 2 months ago

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.