search

ah- / rdkafka-dotnet

C# Apache Kafka client
Other
242 stars 73 forks source link

consumer not reading old messages #53

Closed toniralpatel closed 8 years ago

toniralpatel commented 8 years ago

Partition count = 6, replication factor = 1, number of consumer = 1 and cluster of 3 node.

I pushed 100 messages on my topic x and kill my application after that. Now I start my consumer consuming from the same topic x and it's not reading any messages.

But, when I leave my consumer application active and pushing the messages it reads all of them as expected.

do you know if I am missing something here... I used this properties. Also, my EnableAutoCommit = true. this.consumerConfig["topic.metadata.refresh.interval.ms"] = "0"; this.consumerConfig["auto.commit.interval.ms"] = "0"; this.consumer = new EventConsumer(consumerConfig, Config.HostIps);

TIA!

ah- commented 8 years ago

Could you try setting auto.offset.reset to earliest? By default it starts consuming from the end if you have never committed any offsets.

toniralpatel commented 8 years ago

thanks for the quick reply... I tried setting up the properties and it says "No such configuration property: auto.offset.reset" here is my code look like

this.consumerConfig = new Config() { GroupId = this.Config.ConsumerGroupName, EnableAutoCommit = true }; this.consumerConfig["auto.commit.interval.ms"] = "0"; this.consumerConfig["auto.offset.reset"] = "earliest"; this.consumer = new EventConsumer(consumerConfig, Config.HostIps);

toniralpatel commented 8 years ago

I found the way to set the "auto.offset.reset" property via TopicConfig and assigning that to DefaultTopicConfig in consumer configuration... but my issue is still there... is not reading the messages that are pushed when consumer was not active.