search

databricks / iceberg-kafka-connect

Apache License 2.0
219 stars 49 forks source link

Add Configurable Deserialization Iceberg Table #183

Open Buktoria opened 10 months ago

Buktoria commented 10 months ago

Add an option to configure connectors with a deserialization table. The following would be expected to happen,

  1. Connector attempts to deserialize the message
  2. If the connector fails it does the following
  3. The connector looks at if a deserialization table is configured. If so then,
  4. Construct a record with the [cluster, topic, partition, offset, byte-array-of-message]
  5. Write the record to the configured deserialization table
  6. Move on to the next message

The motivation for having this is to prevent the connector from getting stuck in the case that it encounters a bad message. These "bad messages" could then be handled manually at a later date.

skushwaha-mdsol commented 8 months ago

@bryanck @Buktoria any updates on this enhancement?