search

bsm / sarama-cluster

Cluster extensions for Sarama, the Go client library for Apache Kafka 0.9 [DEPRECATED]
MIT License
1.01k stars 222 forks source link

initial messages in partitions are not delivered while sarama-cluster client was offline #263

Closed notmaxx closed 6 years ago

notmaxx commented 6 years ago

hi,

I just started playing with sarama-cluster and discovered interesting behaviour:

  1. there is topic with 32 partitions
  2. producer wrote to the some partitions only (no consumer running)
  3. I ran sarama-cluster consumer with the group name jh_group and producer generated several messages while consumer was up and running, consumer successfully received them and printed
  4. i stopped consumer and producer sent more messages (like 10 or so) but only part of them is shown as LAG (2+4) 
    
$ kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group jh_group
Consumer group 'jh_group' has no active members.

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID jh_development 10 2 2 0 - - - jh_development 14 2 2 0 - - - jh_development 12 4 4 0 - - - jh_development 5 2 2 0 - - - jh_development 28 2 2 0 - - - jh_development 9 4 4 0 - - - jh_development 22 4 6 2 - - - jh_development 26 4 4 0 - - - jh_development 17 2 6 4 - - - jh_development 19 2 2 0 - - - jh_development 24 2 2 0 - - - jh_development 31 4 4 0 - - - jh_development 15 4 4 0 - - - jh_development 25 4 4 0 - - -

5. I started consumer and offsets description changed

$ kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group jh_group

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID jh_development 0 - 6 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 10 2 2 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 23 - 5 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 1 - 6 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 14 2 2 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 12 4 4 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 5 2 2 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 28 2 2 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 30 - 6 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 9 4 4 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 18 - 4 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 8 - 6 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 29 - 4 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 4 - 4 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 22 6 6 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 26 4 4 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 16 - 6 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 7 - 6 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 13 - 2 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 17 6 6 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 19 2 2 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 24 2 2 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 2 - 2 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 27 - 2 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 21 - 4 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 31 4 4 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 15 4 4 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 20 - 2 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 6 - 8 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 25 4 4 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 3 - 4 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 11 - 4 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama

As you can see above current-offset is `-` for partitions from which we never read messages while consumer was online. **So I have no idea how to pull these messages.** I also marked partition 6 on the snippet above.

6. I sent couple messages while consumer was online and some of them came to partitions from which we never read before and I only received that last messages (not those that were not delivered on this partition before) but offset was marked so messages are commited

$ kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group jh_group

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID jh_development 0 - 6 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 10 2 2 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 23 - 5 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 1 - 6 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 14 2 2 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 12 4 4 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 5 2 2 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 28 2 2 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 30 - 6 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 9 4 4 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 18 - 4 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 8 - 6 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 29 - 4 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 4 - 4 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 22 6 6 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 26 4 4 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 16 - 6 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 7 - 6 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 13 - 2 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 17 6 6 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 19 2 2 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 24 2 2 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 2 - 2 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 27 - 2 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 21 - 4 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 31 4 4 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 15 4 4 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 20 - 2 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 6 10 10 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 25 4 4 0 sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 3 - 4 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama jh_development 11 - 4 - sarama-85bba1d2-f963-4e9f-b819-240856837ed5 /10.10.8.110 sarama


As consumer I use consumer code from example https://github.com/bsm/sarama-cluster#examples

So how can I get the initial messages from partitions that were sent while consumer was offline?

My expectations according to this https://github.com/bsm/sarama-cluster/issues/106 was that I get all messages even when consumer was offline
dim commented 6 years ago

The will be if you change sarama's config default from OffsetNewest to OffsetOldest - https://github.com/Shopify/sarama/blob/master/config.go#L331