FundingCircle / jackdaw

A Clojure library for the Apache Kafka distributed streaming platform.
https://fundingcircle.github.io/jackdaw/
BSD 3-Clause "New" or "Revised" License
369 stars 80 forks source link

Add kafka message header support for test machine #260

Closed 99-not-out closed 4 years ago

99-not-out commented 4 years ago

Adds basic support to test machine for reading and writing message headers. Unfortunately, the Kafka Rest API does not appear to support message headers so its not available there - however the broker and mock transports do support headers.

This is deliberately a light touch Test Machine only change to allow testing and experimentation with the support of headers more widely in Jackdaw, or any topologies tested via Test Machine. No attempt at serdes support is added here, headers must be provided in the form the Java API expects which is a map of:

String k -> byte[] value

The support does at least let you provide them in a clojure Map. Writes are supported via an additional option for the :write! command, e.g.:

[:write! "my-topic" {:id 1 :message "yolo"} {:headers {"VERSION" (.getBytes "1.0.0")}]

Any headers read into the journal for a test are captured in the top level structure for each message in the journal, e.g. for the above write you would see:

{:topic "my-topic",
 :key 1,
 :value  {:id 1, :message "yolo"},
 :partition 0,
 :offset 0,
 :headers {"VERSION" ...byte array...}}  <<-- headers added here

It is left to the user to encode and decode the byte[]'s used for headers (as per the Kafka API).

See test-write-then-read-with-headers in test/jackdaw/test_test.clj for an end to end example.

codecov[bot] commented 4 years ago

Codecov Report

Merging #260 into master will increase coverage by 0.29%. The diff coverage is 73.33%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #260      +/-   ##
==========================================
+ Coverage   80.87%   81.16%   +0.29%     
==========================================
  Files          41       41              
  Lines        2562     2586      +24     
  Branches      149      153       +4     
==========================================
+ Hits         2072     2099      +27     
+ Misses        341      334       -7     
- Partials      149      153       +4     
Impacted Files Coverage Δ
src/jackdaw/test/transports/mock.clj 81.74% <50.00%> (-2.74%) :arrow_down:
src/jackdaw/test/transports/kafka.clj 85.61% <81.81%> (-0.44%) :arrow_down:
src/jackdaw/test/commands/write.clj 93.18% <100.00%> (+0.32%) :arrow_up:
src/jackdaw/test/serde.clj 94.59% <100.00%> (+0.07%) :arrow_up:
src/jackdaw/test/transports/rest_proxy.clj 84.68% <100.00%> (+0.07%) :arrow_up:
src/jackdaw/serdes/edn2.clj 90.90% <0.00%> (+45.45%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 6b43dc9...716b394. Read the comment docs.

marcoccchan commented 4 years ago

Can we document the new :headers option in https://github.com/FundingCircle/jackdaw/blob/8cbc24d89fef2219a9fc0d7dc94a04fceac4cbf0/src/jackdaw/test/commands.clj#L80-L86?