record headers does not work with fetch v3

[yingyingtang-brex]

Hey kafka_ex & kafka experts, I have a question regarding the record header support.

With the new release of kayrock and kafka_ex, I understand that now it support record headers.

I tried to produce records with headers, however I noticed that the fetched message, when I was using fetch version v3 (by setting api_versions: %{fetch: 3}), was with old format (the magic value is 1 when I printed here). I then tried to update the fetch version to v5, and with the same produced records, the fetched message has the new format (magic value is 2) and the header value can be obtained.

I thought fetch v3 should support RecordBatch, not sure why I need to update fetch version to v5 in order to get the latest message format. Also, I noticed in your record_batch_test.exs, the fetch v5 has header test case, while v3 only has the old message test case, I tried to change this v3 test case to produce record batch the same as the v5 test case, but failed with error message in the test docker kafka broker: java.lang.IllegalArgumentException: Magic v1 does not support record headers

Does it mean fetch v3 does not support new format?

[joshuawscott]

According to the Kafka docs https://kafka.apache.org/0110/documentation.html#upgrade_1100_new_protocols

• [KIP-107](https://cwiki.apache.org/confluence/display/KAFKA/KIP-107%3A+Add+purgeDataBefore()+API+in+AdminClient): FetchRequest v5 introduces a partition-level log_start_offset field.
• [KIP-107](https://cwiki.apache.org/confluence/display/KAFKA/KIP-107%3A+Add+purgeDataBefore()+API+in+AdminClient): FetchResponse v5 introduces a partition-level log_start_offset field.
• KIP-82: ProduceRequest v3 introduces an array of header in the message protocol, containing key field and value field.
• KIP-82: FetchResponse v5 introduces an array of header in the message protocol, containing key field and value field.

So to use headers, the minimum API versions are v3 Produce and v5 Fetch.