search

ClickHouse / ClickHouse

ClickHouse® is a real-time analytics DBMS
https://clickhouse.com
Apache License 2.0
37.77k stars 6.93k forks source link

Iceberg Integration: Support compressed manifest files #70874

Open netapp-vaughan opened 1 month ago

netapp-vaughan commented 1 month ago

Use case

Our Iceberg catalog provider enables compression of the manifest.json file by default. The current Clickhouse Iceberg integration makes it impossible to open this file.

Describe the solution you'd like

Clickhouse can read compressed Iceberg manifest files where write.metadata.compression-codec = gzip is set on the table.

Describe alternatives you've considered

A workaround is to disable the compression of the manifest file by setting write.metadata.compression-codec = none

Additional context

https://iceberg.apache.org/docs/latest/configuration/#write-properties

melvynator commented 3 weeks ago

@netapp-vaughan What catalog are you using?

netapp-vaughan commented 3 weeks ago

@netapp-vaughan What catalog are you using?

REST Catalog.