search

nats-io / nats-kafka

NATS to Kafka Bridging
Apache License 2.0
131 stars 32 forks source link

Too many slow consumer error #33

Closed iercan closed 3 years ago

iercan commented 3 years ago

Hi. We are using NatsToKafka connector to relay data from nats to kafka. The subject we are trying to read is sending about 10k messages per second. In nats-kafka logs, we are seeing too many of the messages below:

2021/04/12 08:09:55.192740 [WRN] nats error nats: slow consumer, messages dropped
2021/04/12 08:09:55.193898 [WRN] nats error nats: slow consumer, messages dropped
2021/04/12 08:09:55.196952 [WRN] nats error nats: slow consumer, messages dropped
2021/04/12 08:09:55.199502 [WRN] nats error nats: slow consumer, messages dropped
2021/04/12 08:09:55.212046 [WRN] nats error nats: slow consumer, messages dropped
2021/04/12 08:09:55.223454 [WRN] nats error nats: slow consumer, messages dropped

I did some research and see it is suggested to tune batch.size and linger.ms for kafka producers but I couldn't figure out how to set these parameters on nats-kafka. I'd appreciate with any suggestion.

variadico commented 3 years ago

Hey, ieran. Currently, nats-kafka only exposes a limited set of configuration, documented here: https://github.com/nats-io/nats-kafka/blob/master/docs/config.md

However, I think we would be open to exposing more options.

We're using Shopify/sarama under the hood as the client library for Kafka.

Searching for batch.size and linger.ms in Sarama, I've found these two relevant issues:

iercan commented 3 years ago

@variadico We have found what causes the issue. Document states that nats queue should be supplied with natsqueue keyword but we investigated code and realized that it should be supplied as queuename keyword. As a result our load balancing does not work properly.