search

apache / camel-kafka-connector-examples

Apache Camel Kafka Connector Examples
https://camel.apache.org
64 stars 43 forks source link

Minio-Connector example with deleteAfterRead, retrieve only once and consume large message size. #313

Closed Ihjaz closed 3 years ago

Ihjaz commented 3 years ago

Hi,

I'm trying to replicate objects in minio from one datacenter to other via kafka using the minio source and sink connectors.

While replicating I do not want the object to be deleted from source minio (deleteAfterRead=false) and I also don't want the object to be retrieved again and again in every poll. Can we have an example configuration to achieve this?

Also if the object is of size 20MB , Is there a way to split them into multiple 1 MB size messages and then retrieve or do I have to increase the max.request.size to retrieve it?

oscerd commented 3 years ago

Have a look at https://camel.apache.org/camel-kafka-connector/latest/idempotency.html for avoid reconsuming the same object.

There is no splitting support and there is no plan to add it in the short term.

If you need to do complex logic, have a look at plain camel. You can create a route consuming from minio, split it by line and send the data to other endpoint