search

Yolean / kubernetes-kafka

Kafka cluster as Kubernetes StatefulSet, plain manifests and config
Apache License 2.0
1.84k stars 733 forks source link

Latency in kafka producer #111

Open arunkjn opened 6 years ago

arunkjn commented 6 years ago

Hi guys, I deployed kafka from https://github.com/Yolean/kubernetes-kafka/tree/v2.1.0 on k8s 1.7 I noticed that when my pod(kafka producer) wants to send a message to kafka broker it takes a lot of time (>1minute). Have any of you experienced such a thing? Or any idea what might be causing this?

solsson commented 6 years ago

This is a very generic kafka problem. I'd be very interested to hear if the setup in this repo causes problems in any way, but to do the analysis you probably have to consult Kafka docs or other Kafka resources. I recommend the book Kafka the Definitive Guide as it contains lots of insights on monitoring etc. It could of course be a Kubernetes problem instead. For example saturated nodes, slow volumes or pod networking issues.

arunkjn commented 6 years ago

Agree that it seems like a generic kafka issue not with this manifest itself. I was not able to figure out what the issue could be. Was trying to gather leads on where should I look. I can assure that we have sufficiently large nodes which are not resource starved. Also for volumes we are mounting gp1 SSD's from ec2 so should be good enough. I am using a rancher based cluster and installed kafka in a separate namespace. My pod is trying to access kafka from a different namespace. BTW if we connect to an independent kafka installation outside the cluster (on dedicated ec2 machines) it works fine.

solsson commented 6 years ago

What client (lib) are you using. Is there any debug loggin, such as problems connecting to brokers?