search

databricks / iceberg-kafka-connect

Apache License 2.0
220 stars 49 forks source link

kafka message metadata #292

Closed manojmukkamala closed 2 months ago

manojmukkamala commented 2 months ago

Hello, How can I get the kafka message metadata e.g. partition, offset, timestamp in to the iceberg table? Thanks!

manojmukkamala commented 2 months ago

Figured out!

"transforms": "InsertKafkaMetadata",      

"transforms.InsertKafkaMetadata.type": "org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.InsertKafkaMetadata.topic.field": "topic",
"transforms.InsertKafkaMetadata.partition.field": "partition",
"transforms.InsertKafkaMetadata.offset.field": "offset",
"transforms.InsertKafkaMetadata.timestamp.field": "timestamp"