[Feature][Connector-v2 Iceberg] need small Files compaction feat

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.

https://seatunnel.apache.org/

Apache License 2.0

7.8k stars 1.75k forks source link

[Feature][Connector-v2 Iceberg] need small Files compaction feat #7162

Open jtchen-study opened 2 months ago

jtchen-study commented 2 months ago

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description

Iceberg need small Files compaction feat， otherwise, querying would be a disaster

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

sunxiaojian commented 1 month ago

Iceberg is an independent action to handle small file compaction, I can try to directly add file compaction in the sink.

github-actions[bot] commented 3 days ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.