Open Arcoth opened 1 year ago
Hi, Arcoth,
Could you please help show some demo code about how to trigger the issue?
According to my understanding, calling the consumser.subscribe(...)
twice (for 2 different topics) would have a chance to reproduce it, right?
Hi,
consider the following scenario: After subscribing and seeking into some topic X, while performing the subscribe on the
KafkaConsumer
of a further topic Y, we end up insiderd_kafka_q_serve
(which polls and executes operations) with an operation ofrko_type ==RD_KAFKA_OP_FETCH
andcb_type == RD_KAFKA_Q_CB_CALLBACK
(presumably concerning topic X). This eventually leads us intord_kafka_poll_cb
which saysand we eventually hit the assertion in the q_serve because all ops must be handled:
The solution I tried was adding the consume_cb that is missing above, which KafkaConsumer does not set. So I add
into KafkaConsumer::registerConfigCallbacks. Interestingly, because the rko version of the fetch operation is outdated, it is discarded anyway, see rd_kafka_consume_cb
so actually the CB is never invoked, however is required by librdkafka logic. Can we add a vacuous callback (perhaps with either an exception or just no-op) so this failure mode is averted?