Closed rukai closed 3 months ago
Comparing rukai:kafka_fix_acks0
(f0f528d) with main
(3cdefda)
⚡ 1
improvements
✅ 36
untouched benchmarks
Benchmark | main |
rukai:kafka_fix_acks0 |
Change | |
---|---|---|---|---|
⚡ | encode_request_produce |
25.9 µs | 22.2 µs | +16.51% |
The encode_request_produce
microbenchmark above slightly improved performance since acks=0 no longer sends the header info to the decoder.
Discovery
This managed to avoid being hit because the test cases were running acks=0 last and never sending any more messages down those connections. But while investigating https://github.com/shotover/shotover-proxy/pull/1588 this started showing up in the acks0 test case presumably because the java driver was reusing the connections it sent the acks=0 produce requests down.
Background info
Codecs are an optional helper mechanism provided by the
tokio_util
crate. They make it easy to attach an implementation of a protocol to a TCP socket. In shotover every database that we support has a unique codec implemented for its protocol. There is no singleCodec
type, instead there are separate read and write codecs, called the Encoder and the Decoder.The kafka protocol does not specify the message type of the responses sent by the kafka broker. Instead it is up to the client to remember the message type of each request it sends out and then match up each response that comes in with its corresponding request so that the response can be correctly decoded.
In order to decode kafka responses, shotover needs the kafka outgoing encoder to instruct the outgoing decoder on how to decode each response. This is done via an mpsc channel, specifically we use the mpsc channel from the standard library.
https://github.com/shotover/shotover-proxy/blob/c1331f8944a1d2b1b005bfd6f728aecfd9b5154c/shotover/src/codec/kafka.rs#L42-L48
The problem
However there is one exception to this that we were not handling! The kafka produce request can set a field named
acks
to determine how many acks the leader broker must receive before responding to the request. As a special case ifacks = 0
then the leader broker will never respond to that request. This is used if latency is the highest concern and lost messages are acceptable.Shotover will never receive a response for a produce with acks=0. So if the encoder tells the decoder to expect a produce response it will expect the next message to be a produce even though that is not the case. This results in attempting to decode responses as the wrong type and therefore failure to decode.
The fix
The correct fix here is to have the encoder skip sending message type information if
acks = 0
. Specifically that is done by checking the result of theMessage::response_is_dummy()
method which checks if the request will not receive a response, including a check foracks = 0
on kafka messages.Now that the issue is fixed we can enable the failing tests in
kafka_int_tests/test_cases.rs
.