segmentio / kafka-go

Kafka library in Go
MIT License
7.29k stars 759 forks source link

Multiple Consumers/Readers for different topics #1301

Open avngzy opened 2 weeks ago

avngzy commented 2 weeks ago

Hi there, wondering if it is possible to create multiple readers/consumers? ~From this issue https://github.com/segmentio/kafka-go/issues/1200 it looks like only 1 reader is possible, but I couldn't find where in the documentation this was stated.~ Edit: Looks like I misread and 2 readers are allowed, but I'm not sure what I'm doing wrong

I'm trying to have 2 different consumer groups consuming from 2 different topics. I've tried creating 2 readers but keep getting connection issues(Connection refused) to the Broker(Amazon MSK) for the 2nd reader I'm creating. Previously when we only had a single consumer/reader, we were able to connect with no issues. Any references to existing documentations on suggestions on how to do so is appreciated!

Kafka Version

What version of kafka-go are you using? 0.4.40

To Reproduce

Resources to reproduce the behavior:

How I'm initialising reader

func (kReader *SegmentioKafkaReader) NewReader(brokers []string, topic, groupID string) error {
    // Check if its a valid reader config
    rc := kafkaGo.ReaderConfig{
        Brokers:  brokers,
        GroupID:  groupID,
        Topic:    topic,
        Dialer:   kafkaGo.DefaultDialer,
        MaxBytes: 10e6, // 10MB
    }
    if err := rc.Validate(); err != nil {
        errStr := fmt.Sprintf("kafka.NewConsumer(): invalid kafka reader config!, err %v", err)
        log.Printf(errStr)
        return err
    }
    kReader.r = kafkaGo.NewReader(rc)
    return nil
}

Errors I am getting

 Error connecting to kafka broker abc.kafka.us-west-2.amazonaws.com:9092. Error: failed to dial: failed to open connection to abc.kafka.us-west-2.amazonaws.com:9092: dial tcp: lookup abc.kafka.us-west-2.amazonaws.com on 10.3.0.10:53: dial tcp 10.3.0.10:53: connect: connection refused.

Expected Behavior

I should be able to create 2 readers with different consumer groups consuming from different topics

avngzy commented 2 weeks ago

@petedannemann Not sure if you're still active but tagging in hopes you are and know the answer