search

FundingCircle / jackdaw

A Clojure library for the Apache Kafka distributed streaming platform.
https://fundingcircle.github.io/jackdaw/
BSD 3-Clause "New" or "Revised" License
368 stars 80 forks source link

Update versions to 7.x and 3.x #352

Closed vincentjames501 closed 1 year ago

vincentjames501 commented 1 year ago

We'd like to use the newer kafka streams/clients/serializer/etc versions. It also helps with M1/M2 machines as this updates the dependency on [org.rocksdb/rocksdbjni "5.18.4"] which will throw

java.lang.UnsatisfiedLinkError: /private/var/folders/mp/8jbrtl0x21337tmsyfrx72s40000gp/T/librocksdbjni338208019188803407.jnilib: dlopen(/private/var/folders/mp/8jbrtl0x21337tmsyfrx72s40000gp/T/librocksdbjni338208019188803407.jnilib, 0x0001): tried: '/private/var/folders/mp/8jbrtl0x21337tmsyfrx72s40000gp/T/librocksdbjni338208019188803407.jnilib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/private/var/folders/mp/8jbrtl0x21337tmsyfrx72s40000gp/T/librocksdbjni338208019188803407.jnilib' (no such file), '/private/var/folders/mp/8jbrtl0x21337tmsyfrx72s40000gp/T/librocksdbjni338208019188803407.jnilib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

Example of some recent dependencies:

                 [io.confluent/kafka-schema-registry-client "7.3.2" :exclusions [com.fasterxml.jackson.core/jackson-databind]]
                 [io.confluent/kafka-avro-serializer "7.3.2"]
                 [io.confluent/kafka-json-schema-serializer "7.3.2"]
                 [org.apache.kafka/kafka-clients "3.4.0"]
                 [org.apache.kafka/kafka-streams "3.4.0"]
                 [org.apache.kafka/kafka-streams-test-utils "3.4.0"]
                 [org.apache.kafka/kafka-clients "3.4.0"]

When bringing them in transiently (overriding from jackdaw) I get the following when building our uberjar:

Reflection warning, jackdaw/data/consumer.clj:20:3 - call to org.apache.kafka.clients.consumer.ConsumerRecord ctor can't be resolved.
Reflection warning, jackdaw/streams/interop.clj:265:6 - call to method join on org.apache.kafka.streams.kstream.KStream can't be resolved (argument types: org.apache.kafka.streams.kstream.KStream, org.apache.kafka.streams.kstream.ValueJoiner, org.apache.kafka.streams.kstream.JoinWindows, org.apache.kafka.streams.kstream.Joined).
Reflection warning, jackdaw/streams/interop.clj:284:6 - call to method leftJoin on org.apache.kafka.streams.kstream.KStream can't be resolved (argument types: org.apache.kafka.streams.kstream.KStream, org.apache.kafka.streams.kstream.ValueJoiner, org.apache.kafka.streams.kstream.JoinWindows, org.apache.kafka.streams.kstream.Joined).
Reflection warning, jackdaw/streams/interop.clj:314:6 - call to method outerJoin on org.apache.kafka.streams.kstream.KStream can't be resolved (argument types: org.apache.kafka.streams.kstream.KStream, org.apache.kafka.streams.kstream.ValueJoiner, org.apache.kafka.streams.kstream.JoinWindows, org.apache.kafka.streams.kstream.Joined).
Reflection warning, jackdaw/streams/interop.clj:322:5 - call to method process on org.apache.kafka.streams.kstream.KStream can't be resolved (argument types: org.apache.kafka.streams.processor.api.ProcessorSupplier, unknown).
Reflection warning, jackdaw/admin.clj:36:32 - call to method deleteTopics on org.apache.kafka.clients.admin.AdminClient can't be resolved (argument types: unknown).
Reflection warning, jackdaw/admin.clj:36:26 - reference to field all can't be resolved.
Reflection warning, jackdaw/admin.clj:39:33 - call to method describeTopics on org.apache.kafka.clients.admin.AdminClient can't be resolved (argument types: unknown, org.apache.kafka.clients.admin.DescribeTopicsOptions).
Reflection warning, jackdaw/admin.clj:39:27 - reference to field all can't be resolved.