In my system, a KafkaConsumer uses a regex subscription to subscribe to topics in the form of <user_id>.<topic>. A ConsumberRebalanceListener is used to write topic offsets and the aggregated topic states into a local cache. Automated tests create and delete users (i.e. topics).
When new topics are created, everything works fine. The logs show that the subscription changed, currently subscribed partitions are revoked and the partitions of the created topics are added to the set of previously assigned TopicsPartitions.
When a topic is deleted (via a DeleteTopicsRequest), the log shows that the subscription changes ("Updatding subscribed topics to: …"). The subscription no longer contains the deleted topics, which is correct. Afterwards the currently assigned partitions are revoked in ConsumerCoordinator._on_join_prepare ("Revoking previously assigned partitions …"). However, the set of revoked partitions no longer contains the partitions of the deleted topic, because the subscription was already updated.
I would expect that deleted TopicPartitions are also passed to ConsumerRebalanceListener.on_partitions_revoked. Is my expectation wrong or is this a bug?
In my system, a KafkaConsumer uses a regex subscription to subscribe to topics in the form of
<user_id>.<topic>. AConsumberRebalanceListeneris used to write topic offsets and the aggregated topic states into a local cache. Automated tests create and delete users (i.e. topics).When new topics are created, everything works fine. The logs show that the subscription changed, currently subscribed partitions are revoked and the partitions of the created topics are added to the set of previously assigned TopicsPartitions.
When a topic is deleted (via a
DeleteTopicsRequest), the log shows that the subscription changes ("Updatding subscribed topics to: …"). The subscription no longer contains the deleted topics, which is correct. Afterwards the currently assigned partitions are revoked inConsumerCoordinator._on_join_prepare("Revoking previously assigned partitions …"). However, the set of revoked partitions no longer contains the partitions of the deleted topic, because the subscription was already updated.I would expect that deleted TopicPartitions are also passed to
ConsumerRebalanceListener.on_partitions_revoked. Is my expectation wrong or is this a bug?