twmb
commented
4 months ago As of v1.16.0, a fake fetch is injected whenever there are load failures on metadata. On produce, it has always been the case that a load failure bumps the retry count, and eventually records will fail if you have limited retries configured. There is also a warn level log on the producer side for the bump.
Metadata code for this: https://github.com/twmb/franz-go/blob/52a9ebaedf4718ca77692a91fca699383d76dd7c/pkg/kgo/metadata.go#L693-L703
The client continues trying in the face of auth errors because the end user can update ACLs and fix auth errors without restarting the client. I'm not sure it's better to fail hard in the case of application failures. I somewhat expect that auth failures are quick enough to notice too such that this isn't a large problem in practice? What happens if you AddConsumeTopics to a topic before you fix auth failures, but the rest of your application is working fine, and you're producing to and consuming from other topics that are working -- should the application continue working but log that you have something to fix (current behavior) or should suddenly your application crash?
Let me know if you disagree or what you think the behavior should be - closing for now leaning to keep the current behavior.
If metadata cannot be updated, client just retries it forever instead of failing after X retries.
For example, trying to create client against the Kafka with non-permissive ACLs doesn't cause error. Instead of that, library keeps logging the following information: