search

elodina / dropwizard-kafka-http

Apache Kafka HTTP Endpoint for producing and consuming messages from topics
http://www.elodina.net
Apache License 2.0
153 stars 45 forks source link

not all messages with no key are sent. #8

Closed eugpermar closed 9 years ago

eugpermar commented 9 years ago

Hi everyone.

I'm trying to use the dropwizard-kafka-http producer interface. When I produce with:

curl -d "topic=http&key=0&message=hello" http://localhost:8080/message

All goes ok. I execute the command 10 times, and I see 10 new messages in the http topic.

However, I don't need the key argument. If I delete it

    curl -d "topic=http&message=hello" http://localhost:8080/message

and I run 10 times, I only see about 4 or 5 hello, but never 10. Is this an expected behavior?

dmitrypekar commented 9 years ago

Hi eugpermar,

This is definitely not expected behavior and I can't reproduce it locally. All messages should be delivered correctly. One possible reason that I see, is that there could some latency in message delivery and consumer just can't fetch them all in a specified duration (500ms by default). To check this try to use numbered messages (message1, message2, ..) and consume several times or increase the timeout.

If that is not the case, then I think that the issue is somehow related to your environment. In that case, please describe your environment in details. Are you:

  1. using single broker or a cluster;
  2. what kind of topic configuration are you using (how many partitions / replicas);
  3. what kind of consumer do you use to check that messages are delivered;
  4. what is the timeout of that consumer?
eugpermar commented 9 years ago

Hi @dmitrypekar, thanks for your fast response.

It was my failure, the key ->partition balancing was not working as I though they were. All ok now, sorry for the invalid issue.