search

FundingCircle / jackdaw

A Clojure library for the Apache Kafka distributed streaming platform.
https://fundingcircle.github.io/jackdaw/
BSD 3-Clause "New" or "Revised" License
369 stars 80 forks source link

Update Jackdaw to work with newer versions of kafka-streams / kafka-streams-test-utils #290

Closed LouiseKlodt closed 3 years ago

LouiseKlodt commented 3 years ago

Currently, the jackdaw.test namespace requires you to import TopologyTestDriver for test-machine to work, which in turn requires you to bring in the org.apache.kafka/kafka-streams-test-utils library as a dependency.

This currently only works using a kafka-streams-test-utils version within 2.0.0 - 2.3.0. When adding versions higher than 2.3.0, it throws errors. For example, for version 2.4.0:

Syntax error (ClassNotFoundException) compiling new at (jackdaw/test.clj:250:5).
org.apache.kafka.streams.internals.QuietStreamsConfig

Errors are thrown for version numbers from 2.4.0, 2.5.0 ... up to the latest: 2.8.0 - they all occur in jackdaw/test.clj.

-> Jackdaw needs to be made compatible / brought up to date with the newest versions from Apache Kafka's libraries.

cddr commented 3 years ago

I think the only thing I needed to do to get it working on later versions of kafka (other than updating dependencies) was to ensure new-topic cast ints to shorts which is required by upstream API now.

https://github.com/cddr/test-machine/compare/75bcf2e..cfab170#diff-c1a95b28cd27156ef869335927cd2bdf20921d544980cf42a11d0740cde7f5a5R22

https://github.com/cddr/test-machine/compare/cfab170..75bcf2e

gphilipp commented 3 years ago

Thanks Andy, we're going to have a look next week.

gphilipp commented 3 years ago

🙋‍♂️ @minimal is going to have a look at this.