Received event limit when using flume-ng-redis

[lueliu]

Hi I have 1277 events/s from redis cluster in Pub/Sub mode to flume redis sub source. But just 2 min,data link will break. Is something wrong? or some by limited?

when 400 events/s is ok.

( redis cluster -----> flume redis source -----> kafka channel )

[chiwanpark]

Hi, Sorry for late reply. I was busy in weeks.

There is no limit in our plugin. I think there is something wrong. Could you upload your flume configuration if it is okay?

[lueliu]

THX

My Flume config:

agent6.sources = r1 agent6.channels = c1

agent6.sources.r1.type = com.chiwanpark.flume.plugins.RedisSubscribeDrivenSource agent6.sources.r1.redisHost = 192.168.0.1 agent6.sources.r1.redisPort = 16001 agent6.sources.r1.redisChannel = gps_channel agent6.sources.r1.redisTimeout = 300000 agent6.sources.r1.messageCharset = utf-8 agent6.sources.r1.channels = c1

agent6.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel agent6.channels.c1.capacity = 1000000 agent6.channels.c1.transactionCapacity = 20000 agent6.channels.c1.brokerList = 192.168.0.2:9092,192.168.0.3:9092,192.168.0.4:9092 agent6.channels.c1.topic = gps-channel agent6.channels.c1.zookeeperConnect = 192.168.0.5:2181,192.168.0.6:2181,192.168.0.7:2181

[lueliu]

your plugin is ok. because I was tested 1277 messages/s with flume memory channel is ok. while I also use 4 source (400 messages/s per source) with flume kafka channel (one topic) is ok.

[chiwanpark]

Hi, Thanks for investigation. If there is no problem in the plugin, please close this issue. :)