search

apache / iceberg-python

Apache PyIceberg
https://py.iceberg.apache.org/
Apache License 2.0
307 stars 113 forks source link

Implement rolling manifest-writers #596

Open Fokko opened 2 months ago

Fokko commented 2 months ago

Feature Request / Improvement

Once we get support for merge-append https://github.com/apache/iceberg-python/pull/363 operations, and also compaction in the future, it would be good to have rolling-manifest writers that open a new file once the manifest gets to a certain size (both bytes and number of entries). This makes sure that Iceberg tables have decent parallelization when doing query planning.

felixscherz commented 2 months ago

Hi, I would like to give this a shot if no one else is currently working on this:) I see the java implementation has a RollingManifestWriter so I'd start there.