How to properly reconsume messages after their processing failed in High-Level balanced consumer?

[fdr400]

Discussed in https://github.com/confluentinc/librdkafka/discussions/4722

^{Originally posted by **fdr400** May 17, 2024} Hello everyone! I am currently implementing the driver for Kafka in @userver-framework. The code is available here: https://github.com/userver-framework/userver/tree/develop/kafka It provides an interface for processing message batches received from the topics consumer has subscribed on. Common use-case of consumer looks like this: ```cpp consumer.Start([](kafka::MessageBatchView batch) { /* message batch processing */ consumer.AsyncCommit(); // wrapper for rd_kafka_commit(consumer_handle, 1) }); ``` `kafka::MessagesBatchView` is an alias for `std::span`, where `kafka::Message` is a C++ wrapper for `rd_kafka_message_t*` The library periodically polls the message batches and invokes the user-provided callback passing the message batch view. I am trying to support the message reconsumption when the callback throws an exception. To implement such logic, current implementation closes the consumer and subscribes for the same topics, after catching the exception, to start reading the topic partition from the last committed offset. But the implementation looks not optimal, because, for example, when using an EAGER rebalance strategy, after closing the consumer, all partitions are firstly revoked from all consumers in the current group and then assigned to them again. I read lots of issues in the repo and googled a lot, but, unfortunately, did not find the, how to reread not committed messages using the **balanced** consumer. I saw several solutions to reread the messages, but, as I understand, the won't work when consumer **subscribed** to topics: 1. First solution is to fetch the current consumer assignment with `rd_kafka_assignment` and assign it to consumer with `rd_kafka_assign`. But in many issues I read that `rd_kafka_assign` assigns the static set to consumer and can not be used outside of rebalance_callback 2. Second, is to use the `rd_kafka_seek` to move the fetched offsets back to the commited values. But, again, I read that seek can used only after manual partition assignment with `rd_kafka_assign` 3. The last solution is to cache the unprocessed message batch to pass it again and again to the use callback, until the callback invokation succeeded. But in such solution consumer stops the polling loop and may be kicked from the consumer group after some time So, the question is, how properly reconsume the uncommitted messages using the balanced consumer (`rd_kafka_subscribe`), with minimum overhead?

[emasab]

Hi @fdr400 this is the correct thing to do

Second, is to use the rd_kafka_seek to move the fetched offsets back to the commited values. But, again, I read that seek can used only after manual partition assignment with rd_kafka_assign

It's possible even when subscribing, given you seek only partitions that are assigned to you, that you can check with rd_kafka_assignment. You should use rd_kafka_seek_partitions to seek.

Check out this example: https://github.com/confluentinc/librdkafka/blob/7fc5a59826ee1e1dee3236b1e43180cc22cfb496/examples/transactions.c#L166

You should probably need also the rebalance_cb to check if partitions are revoked from your member. That callback is called on consume, so after processing your last consumed message, but partition isn't revoked until you call assign(NULL) or incremental_unassign() inside that one.

[fdr400]

@emasab thanks for your answer! I'll try to do so. But it is not obvious from documentation, because rd_kafka_seek* functions are in Simple Consumer API group