mfontanini / cppkafka

Modern C++ Apache Kafka client library (wrapper for librdkafka)
BSD 2-Clause "Simplified" License
600 stars 206 forks source link

How to escape offline messages to Consumer? #243

Closed Ryan-Lew closed 4 years ago

Ryan-Lew commented 4 years ago

Consumer subscribes topics. Consumer recieve many offline messages when reconnect to Kafka server.How to escape the offline messages?I have tested by setting topic config "auto.offset.reset" to "latest" and "smallest", but no effect.

mfontanini commented 4 years ago

Are you committing any offsets when you receive messages? That's the way to say "I acknowledge this message (and every other before it in this partition) and don't want to see it again".

accelerated commented 4 years ago

@Ryan-Lew, you can set auto.offset.reset=latest and also when you assign your partitions on startup by calling assign() (either in the rebalance callback if you use dynamic partition assignment via subscribe) or by calling it directly (static assignment), set the starting offset to TopicPartition::OFFSET_END for all your partitions. Also you must commit each message afterwards as indicated above.

Ryan-Lew commented 4 years ago

@Ryan-Lew, you can set auto.offset.reset=latest and also when you assign your partitions on startup by calling assign() (either in the rebalance callback if you use dynamic partition assignment via subscribe) or by calling it directly (static assignment), set the starting offset to TopicPartition::OFFSET_END for all your partitions. Also you must commit each message afterwards as indicated above.

Thanks! So, I have to assign my partitions by calling 'assign()' not 'subscribe()'?

accelerated commented 4 years ago

Subscribe doesn't have offsets AFIK. So yes, just using assign.

Ryan-Lew commented 4 years ago

ok,thank you.