databricks / iceberg-kafka-connect

Apache License 2.0
219 stars 49 forks source link

partition-by with transform #307

Closed fuzing closed 3 weeks ago

fuzing commented 3 weeks ago

Greetings......

There doesn't appear to be a way to include an iceberg transform as part of the partition-by field/s.

For example, iceberg allows us to use the following field-spec as part of a partition-spec: { 'source-id': 1, 'name': 'timestamp', 'transform': 'day' }

It would be great if this connector would allow these transforms to be included as part of the partition-by directive - for example, one might use the following syntax: iceberg.table.<table name>.partition-by=day(timestamp) and iceberg.tables.default-partition-by=day(timestamp)