emasab commented 3 months ago

Metadata cache by topic id is added, with an AVL tree having a Uuid as key and the same struct as value.

Metadata cache was cleared on full metadata refresh, leading to unnecessary refreshes and occasional UNKNOWN_TOPIC_OR_PART errors. Solved by updating cache for existing or hinted entries instead of clearing them. Happening since 2.1.0.
A metadata call before member joins consumer group, could lead to an UNKNOWN_TOPIC_OR_PART error. Solved by updating the consumer group following a metadata refresh only in safe states. Happening since 2.1.0.
Metadata refreshes without partition leader change could lead to a loop of metadata calls at fixed intervals. Solved by stopping metadata refresh when all existing metadata is non-stale. Happening since 2.3.0.
A partition migration could happen, using stale metadata, when the partition was undergoing a validation and being retried because of an error. Solved by doing a partition migration only with a non-stale leader epoch. Happening since 2.1.0.

emasab commented 3 months ago

Closes #4577

confluentinc / librdkafka

4676