Open ctron opened 3 years ago
Yes, this is a known limitation that will not be fixed. You will need a group.id configured to be able to commit or retrieve committed offsets, and if not the group.id is not really used so you can just generate a unique string (e.g., uuid).
This was a limitation of the Java client until recently: https://github.com/apache/kafka/pull/7943
I would say this is worth changing in librdkafka because it's pretty high visibility and does shift perceptions negatively because it doesn't behave in the way people expect. "I don't want to use a consumer group - WTF!" was my reaction to this when I was new to librdkafka. I may not have considered committing offsets, can't recall, but i may not have needed it and the more natural time / time with less friction / most impactful time to learn about that would have been at the point where I tried to do it.
It also doesn't look hard to change in the code.
If we allow group.id to not be set it means commits will not work, so should we then automatically disable enable.auto.commit? That will trip people up since we would disable something they might rely on working. So instead we could require enable.auto.commit to be configured to false when group.id is not configured, but then we're pretty much back to where we started with users needing to understand how these things fit together.
I'm not super convinced this niche use-case needs to be solved.
I do think that would be better (forcing enble.auto.commit to be false) - because the error would be informative - would teach me something i didn't know rather than leave me wondering. In the Java case, it appears they just disable auto commit (seems ok - i don't like defaults conditionally changing, but we do it elsewhere - you are right though it is potentially confusing). What would be even imo better is if enable.auto.commit defaulted to false (but that can't really be changed at this point). Especially as I don't like auto commit because it gives neither at-least-once or at-most-once semantics.
Okay, so:
I guess we should also prohibit RD_KAFKA_OFFSET_STORED as starting offset in seek(), assign(), etc when group.id is not set.
Hi guys. What's the status of this issue? Was it fixed by any chance? Having the same problem when using assign()
Hi guys. Is there any update on this issue?
I think this behavior is defined in KIP-289: Improve the default group id behavior in KafkaConsumer (Instead of https://github.com/apache/kafka/pull/7943, which just changed null
to None
IMO)
INTRODUCTION.md mentioned KIP-289 is implemented, but it seems not the case.
BTW, if you don't want consumer groups being unnecessarily created, I think you can just disable enable.auto.commit
, and specify an arbitrary group.id
. In this case it doesn't seem to have any effect.
Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ
Description