consumer group not always used?

[ScottChapman]

We are running logstash 1.5beta1 (0.8.2 kafka) and running about 60 logstashes, all using derivative configs where we specify the consumer group to be something like "ProductionLogstash".

All our topics are single partition topics.

However when we run the offset checker we only see a couple of offsets being tracked for that consumer group.

Any ideas?

[joekiller]

Could you give two example configurations and the offset command you are running? On Feb 17, 2015 11:26 AM, "ScottChapman" notifications@github.com wrote:

We are running logstash 1.5beta1 (0.8.2 kafka) and running about 60 logstashes, all using derivative configs where we specify the consumer group to be something like "ProductionLogstash".

All our topics are single partition topics.

However when we run the offset checker we only see a couple of offsets being tracked for that consumer group.

Any ideas?

— Reply to this email directly or view it on GitHub https://github.com/joekiller/logstash-kafka/issues/50.

[ScottChapman]

Yep. Here is what the kafka section of one of our configs looks like: input { kafka { reset_beginning => "true" zk_connect => "localhost:3011,localhost:3012,localhost:3013" topic_id => "log_ourhost1_heartbeat" group_id => "logstash-test"

    type => "logmonitor"
    codec => multiline {
        pattern => "^(%{LOGMONITORTIME} - |%{NUMBER}:)"
        negate => true
        what => previous
        patterns_dir => "config/patterns"
        charset => "ASCII-8BIT"
    }
}

}

What we are seeing for offsets for that group is: Group Topic Pid Offset logSize Lag Owner logstash-test log_ourhost2_sugarcrm 0 115309 115309 0 logstash-test_devfp02-1424186288992-80b78ff5-0 logstash-test log_ourhost3_sugarcrm 0 5514 5514 0 logstash-test_devfp02-1424186288883-1dccf6a2-0

[ScottChapman]

Just FYI, if we use a unique group-id for each topic (and thus unique for each logstash instance) it all works. It just appears is you have multiple logstash instances running (different configs) using the same group-id we see bad things...

[joekiller]

I think the reset-beginning toggle is the problem here. There is a bug where this'll reset the consumer group regardless of what others in the group are doing. I have it partially fixed in jruby-kafka and recently got the logstash guys to fix a bug preventing tests from working with the latest library. For now I'd suggest not using the reset-beginning toggle. On Feb 17, 2015 5:32 PM, "ScottChapman" notifications@github.com wrote:

Just FYI, if we use a unique group-id for each topic (and thus unique for each logstash instance) it all works. It just appears is you have multiple logstash instances running (different configs) using the same group-id we see bad things...

— Reply to this email directly or view it on GitHub https://github.com/joekiller/logstash-kafka/issues/50#issuecomment-74769389 .

[joekiller]

I've fixed up reset-beginning somewhat but it is still destructive. The docs better reflect what it actually does now. Please re-open if you have problems otherwise.