awslabs / kinesis-kafka-connector

kinesis-kafka-connector is connector based on Kafka Connect to publish messages to Amazon Kinesis streams or Amazon Kinesis Firehose.
Apache License 2.0
153 stars 91 forks source link

Is it possible to change the format of the data being sinked to kinesis? #73

Open miguellgramacho96 opened 1 year ago

miguellgramacho96 commented 1 year ago

Hi!

Wondering if it's possible to use the "value.converter" to read avro messages but then use some sort of property (not sure if this is a supported feature) to format the data, to a different format, and then sink it to a kinesis stream?

This is currently supported by a different connector: https://www.confluent.io/hub/confluentinc/kafka-connect-s3, with the "format.class" property.

Thanks!

bdesert commented 1 year ago

format.class in S3 connector is custom implementation and not a part of Kafka Connect framework. This is not implemented in Kinesis connectors. if you would like this to be implemented, please update this issue with initial requirements, so we can convert it to enhancement. if you would like to contribute your implementation, please go ahead and and create PR once done.