Kafka optimizations

[eapache]

@tylertreat I found your blog post after Sarama's traffic spiked (thanks!) - here are two optimizations which should substantially improve kafka throughput (though I don't have an environment set up to verify that at the moment).

It's not clear to me from the post whether the Send() message is expected to be synchronous or not (ie whether it has to wait for a response). If so then the second optimization is invalid, but I would note that asynchronous producing is the "typical" use case of Kafka, so comparing it with other systems based on synchronous performance isn't all that useful.

Feel free to reach out to me with any other questions you might have about Sarama/Kafka.

[tylertreat]

Hi Evan, thanks for the PR. I'm seeing this issue in Kafka when running the default throughput benchmark (1 million messages) using your changes:

Closing socket connection to /127.0.0.1 due to invalid request: Request of length 144846617 is not valid, it is larger than the maximum size of 104857600 bytes. (kafka.network.Processor)

Any ideas?

[eapache]

Oh, heh, that's a new one. Because the benchmark is a tight loop, it's pushing so many messages at once that we're constructing a request larger than the kafka broker will accept by default (request: 144MB, default max: 100MB).

I've never seen this before (nobody tries to push 150MB simultaneously in non-benchmark scenarios) so we don't actually have code in place to limit this. You can bump the kafka configuration value (socket.request.max.bytes) to accommodate for now; the better fix is in Sarama, so please file a ticket there with the relevant details.

CC @wvanbergen this is an interesting one

(edit: and by kafka configuration value, I mean in the kafka broker's config, not in the benchmark anywhere)

[tylertreat]

I increased the max bytes setting (I also had to increase the JVM heap size), but the benchmark doesn't seem to finish with 1M messages. With 100,000 messages, the receiver throughput was about 400 messages/second. Have you had luck with this?

[eapache]

"Doesn't seem to finish" is pretty vague. I will actually have a broker available to play with tomorrow, so I will do some experiments then. If we're constructing one giant packet (as I suspect) then I have no idea what that will do to the latency characteristics, but it's possible something like this might happen.

Tangential note: make sure you have the absolute latest version of Sarama for your testing since we made the default consumer configuration far more performant recently.

[tylertreat]

"Doesn't seem to finish" as in it appears to be deadlocked despite letting it run for ~1 hour. How does Sarama batch messages?

[eapache]

"Appears to be deadlocked" is weird in go, since go's scheduler will panic on any real deadlock. If you Ctrl+\ at that point it should dump a trace of what goroutines are blocked where.

Sarama sends the first message right away, then waits until it receives the ack. When it can send again, it batches all messages it's received in the meantime (which is all 1 million less the first, since the benchmark loop is tight).

[eapache]

Ok, so playing with this a bit, the consumer "slowness" is primarily because, when we send all the messages in a single request, the consumer can't consume anything until they've all been received, so all the time the producer is running, the consumer is wasting. Haven't bothered trying to reproduce your "hang" but I assume it's something similar. You can get better consumer throughput numbers for the 100,000 case by increasing consumerConfig.DefaultFetchSize in sarama (1024*1024 works well for me) but again this is kind of a workaround for the real fix.

I will hack together a message size limit in sarama and see what that does.

[eapache]

Okay, so I've added another commit to this PR, and I've added a message-size limit in sarama (see branch producer-size-limit, not yet merged, PR is https://github.com/Shopify/sarama/pull/137).

Between these two, I now get (without touching the kafka broker or java heap size):

$ go run main.go kafka
2014/07/14 13:19:19 Begin kafka test
2014/07/14 13:19:28 Sent 1000000 messages in 8725.524414 ms
2014/07/14 13:19:28 Sent 114606.289062 per second
2014/07/14 13:19:31 Received 1000000 messages in 11666.221680 ms
2014/07/14 13:19:31 Received 85717.554688 per second
2014/07/14 13:19:31 End kafka test

which is in line with what I'd expect.

[tylertreat]

Yep, this looks really good. I'm seeing similar results. I will update the benchmarks this evening.

[eapache]

Note my original comment regarding whether Send must be synchronous or not still applies.

[tylertreat]

There is no requirement for Send to be synchronous. I do need to revisit other benchmarks to ensure I'm not accidentally enforcing this constraint if the client supports async.