search

apache / camel-kafka-connector

Camel Kafka Connector allows you to use all Camel components as Kafka Connect connectors
https://camel.apache.org
Apache License 2.0
152 stars 100 forks source link

Kafka Events batching #1605

Closed AishD3 closed 4 weeks ago

AishD3 commented 7 months ago

Hi @oscerd , we would like batch some amount of kafka records while a file is created in data lake . We are using below configs in Kafka Connect for that , but batching is not working , is it correct and am I missing something. 

camel.aggregation.size: 5
camel.aggregation.timeout: 5000
oscerd commented 7 months ago

https://camel.apache.org/camel-kafka-connector/4.0.x/user-guide/aggregation.html

AishD3 commented 7 months ago

@oscerd Yes we referred to this , but do we need to add camel.beans.aggregate=#class:org.apache.camel.kafkaconnector.aggregator.StringAggregator

as well , or only

camel.aggregation.size: 5 camel.aggregation.timeout: 5000

is enough

oscerd commented 7 months ago

You need to add a bean because otherwise Camel don't know how to aggregate.

AishD3 commented 7 months ago

Thanks a lot its working

skumaravelan commented 2 weeks ago

Can you share the dsl

skumaravelan commented 2 weeks ago

@AishD3 , could you share the dsl and how touse this aggregator