[X] I searched in the issues and found nothing similar.
Motivation
Currently, the doc of full compaction changelog producer states that "Full compaction changelog producer can produce complete changelog for any type of source", however, when full-compaction.delta-commits is greater than 1, the intermediate changes across multiple snapshots will be ignored.
Iceberg CDC refers to this as net changes, and Snowflake refers to this as Minimum-delta changes, both differ from a "complete" changelog. So I think this also worth clarifying in the Paimon doc, because we usually consider net changes and complete changes to be different.
Solution
I think we should clarify that the full compaction changelog producer will only output complete changes when full-compaction.delta-commits is set to 1; when full-compaction.delta-commits is set to a value greater than 1, intermediate changes across the serveral delta snapshots will be ignored.
Search before asking
Motivation
Currently, the doc of full compaction changelog producer states that "Full compaction changelog producer can produce complete changelog for any type of source", however, when
full-compaction.delta-commitsis greater than 1, the intermediate changes across multiple snapshots will be ignored.Iceberg CDC refers to this as net changes, and Snowflake refers to this as Minimum-delta changes, both differ from a "complete" changelog. So I think this also worth clarifying in the Paimon doc, because we usually consider net changes and complete changes to be different.
Solution
I think we should clarify that the full compaction changelog producer will only output complete changes when
full-compaction.delta-commitsis set to 1; whenfull-compaction.delta-commitsis set to a value greater than 1, intermediate changes across the serveral delta snapshots will be ignored.cc @JingsongLi What do you think?
Anything else?
No response
Are you willing to submit a PR?