[core] Generate changelog by copying data when records are insert-only

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

https://paimon.apache.org/

Apache License 2.0

2.12k stars 839 forks source link

Closed yunfengzhou-hub closed 6 days ago

yunfengzhou-hub commented 2 weeks ago

Linked issue: #3567

Implements custom copy in local filesystem and OSS
Adds the optimization to generate changelog by copying data file
Triggers the optimization when the following conditions are met
- The filesystem implements copy
- The changelog producer is INPUT
- all records received since last flush are insert-only

This PR does not affect API or storage format.

This PR does not add a new feature that requires document.

yunfengzhou-hub commented 1 week ago

Hi @JingsongLi could you please take a look at this PR?

yunfengzhou-hub commented 1 week ago

Thanks for the comments @JingsongLi. I have updated the PR according to the comments.