search

apache / camel-kafka-connector-examples

Apache Camel Kafka Connector Examples
https://camel.apache.org
64 stars 43 forks source link

Example how to export from Kafka to S3 and import back from S3 to Kafka both key and value #322

Closed gohanbg closed 2 years ago

gohanbg commented 3 years ago

Hello,

I think you are doing a great job with all the kafka connectors that you are building!

Some time ago I played with the org.apache.camel.kafkaconnector.awss3.CamelAwss3SinkConnector connector and successfully managed to export records from my topic to S3.

I now want to be able to import the records from S3 back to kafka. That is fairly straight forward, but there is one tricky part - I want to keep the keys.

For example if I had a record with

key="key_UUID"
value="value_JSON"

I want to export both the key and the value, and then be able to create the same record with the same key and same value. Is this possible, and if so can you give me an example configuration for the S3 Sink/Source.

Thanks Mihail

oscerd commented 3 years ago

You can have a look at this SMT

https://github.com/apache/camel-kafka-connector/pull/1204/files

It's still not merged in the code base.

gohanbg commented 3 years ago

Thanks @oscerd.