search

spring-projects / spring-kafka

Provides Familiar Spring Abstractions for Apache Kafka
https://projects.spring.io/spring-kafka
Apache License 2.0
2.16k stars 1.54k forks source link

Kafka_2.13 version in spring-kafka-test is different from the one in spring-boot 2.7.3 #2398

Closed MichaelZett closed 1 year ago

MichaelZett commented 1 year ago

*In what version(s) of Spring for Apache Kafka are you seeing this issue? 2.8.8

Describe the bug Spring-kafka-test 2.8.8 is used in spring-boot 2.7.3. There kafka_2.13 version is 3.1.1 (https://docs.spring.io/spring-boot/docs/2.7.3/reference/html/dependency-versions.html#appendix.dependency-versions). In spring-kafka-test kafka_2.13 version is version is only 3.0.1 - shouldn't that fit, so spring-kafka-test should also use kafka_2.13 version 3.1.1? In one of our projects that leads to a problem. We are able to work-around that by exlicitely use kafka_2.13 version is 3.1.1. Nevertheless we wanted to inform you about the mismatch.

garyrussell commented 1 year ago

Something else must be overriding the version; boot brings in the same version of all kafka libraries...

https://github.com/spring-projects/spring-boot/blob/v2.7.3/spring-boot-project/spring-boot-dependencies/build.gradle#L1055-L1092

Look at the dependency tree to see which is the culprit.

This is what ./mvnw dependency:tree shows for my 28 smoke test using boot 2.7.3

[INFO] com.example:ktest28:jar:0.0.1-SNAPSHOT
[INFO] +- org.springframework.boot:spring-boot-starter-test:jar:2.7.3:test
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.7.3:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.7.3:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.7.3:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.7.3:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.11:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.11:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.17.2:compile
[INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.17.2:compile
[INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.36:compile
[INFO] |  |  +- jakarta.annotation:jakarta.annotation-api:jar:1.3.5:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.30:compile
[INFO] |  +- org.springframework.boot:spring-boot-test:jar:2.7.3:test
[INFO] |  +- org.springframework.boot:spring-boot-test-autoconfigure:jar:2.7.3:test
[INFO] |  +- com.jayway.jsonpath:json-path:jar:2.7.0:test
[INFO] |  |  \- net.minidev:json-smart:jar:2.4.8:test
[INFO] |  |     \- net.minidev:accessors-smart:jar:2.4.8:test
[INFO] |  |        \- org.ow2.asm:asm:jar:9.1:test
[INFO] |  +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.3:test
[INFO] |  |  \- jakarta.activation:jakarta.activation-api:jar:1.2.2:test
[INFO] |  +- org.assertj:assertj-core:jar:3.22.0:test
[INFO] |  +- org.hamcrest:hamcrest:jar:2.2:test
[INFO] |  +- org.junit.jupiter:junit-jupiter:jar:5.8.2:test
[INFO] |  |  +- org.junit.jupiter:junit-jupiter-params:jar:5.8.2:test
[INFO] |  |  \- org.junit.jupiter:junit-jupiter-engine:jar:5.8.2:test
[INFO] |  |     \- org.junit.platform:junit-platform-engine:jar:1.8.2:test
[INFO] |  +- org.mockito:mockito-core:jar:4.5.1:test
[INFO] |  |  +- net.bytebuddy:byte-buddy:jar:1.12.13:test
[INFO] |  |  +- net.bytebuddy:byte-buddy-agent:jar:1.12.13:test
[INFO] |  |  \- org.objenesis:objenesis:jar:3.2:test
[INFO] |  +- org.mockito:mockito-junit-jupiter:jar:4.5.1:test
[INFO] |  +- org.skyscreamer:jsonassert:jar:1.5.1:test
[INFO] |  |  \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test
[INFO] |  +- org.springframework:spring-core:jar:5.3.22:compile
[INFO] |  |  \- org.springframework:spring-jcl:jar:5.3.22:compile
[INFO] |  +- org.springframework:spring-test:jar:5.3.22:test
[INFO] |  \- org.xmlunit:xmlunit-core:jar:2.9.0:test
[INFO] +- org.springframework.boot:spring-boot-starter-integration:jar:2.7.3:compile
[INFO] |  \- org.springframework.boot:spring-boot-starter-aop:jar:2.7.3:compile
[INFO] |     \- org.aspectj:aspectjweaver:jar:1.9.7:compile
[INFO] +- org.springframework.integration:spring-integration-kafka:jar:5.5.14:compile
[INFO] +- org.springframework.integration:spring-integration-core:jar:5.5.14:compile
[INFO] |  +- org.springframework:spring-aop:jar:5.3.22:compile
[INFO] |  |  \- org.springframework:spring-beans:jar:5.3.22:compile
[INFO] |  +- org.springframework:spring-context:jar:5.3.22:compile
[INFO] |  |  \- org.springframework:spring-expression:jar:5.3.22:compile
[INFO] |  +- org.springframework:spring-messaging:jar:5.3.22:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.3.22:compile
[INFO] |  +- org.springframework.retry:spring-retry:jar:1.3.3:compile
[INFO] |  \- io.projectreactor:reactor-core:jar:3.4.22:compile
[INFO] |     \- org.reactivestreams:reactive-streams:jar:1.0.4:compile
[INFO] +- org.springframework.integration:spring-integration-test:jar:5.5.14:test
[INFO] |  \- org.springframework.integration:spring-integration-test-support:jar:5.5.14:test
[INFO] |     \- org.hamcrest:hamcrest-library:jar:2.2:test
[INFO] |        \- org.hamcrest:hamcrest-core:jar:2.2:test
[INFO] +- org.springframework.kafka:spring-kafka:jar:2.8.8:compile
[INFO] |  +- org.apache.kafka:kafka-clients:jar:3.1.1:compile
[INFO] |  |  +- com.github.luben:zstd-jni:jar:1.5.0-4:runtime
[INFO] |  |  +- org.lz4:lz4-java:jar:1.8.0:runtime
[INFO] |  |  \- org.xerial.snappy:snappy-java:jar:1.1.8.4:runtime
[INFO] |  \- com.google.code.findbugs:jsr305:jar:3.0.2:runtime
[INFO] +- org.springframework.kafka:spring-kafka-test:jar:2.8.8:test
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.6.3:test
[INFO] |  |  +- org.apache.zookeeper:zookeeper-jute:jar:3.6.3:test
[INFO] |  |  +- org.apache.yetus:audience-annotations:jar:0.5.0:test
[INFO] |  |  +- io.netty:netty-handler:jar:4.1.79.Final:test
[INFO] |  |  |  +- io.netty:netty-common:jar:4.1.79.Final:test
[INFO] |  |  |  +- io.netty:netty-resolver:jar:4.1.79.Final:test
[INFO] |  |  |  +- io.netty:netty-buffer:jar:4.1.79.Final:test
[INFO] |  |  |  +- io.netty:netty-transport:jar:4.1.79.Final:test
[INFO] |  |  |  +- io.netty:netty-transport-native-unix-common:jar:4.1.79.Final:test
[INFO] |  |  |  \- io.netty:netty-codec:jar:4.1.79.Final:test
[INFO] |  |  \- io.netty:netty-transport-native-epoll:jar:4.1.79.Final:test
[INFO] |  |     \- io.netty:netty-transport-classes-epoll:jar:4.1.79.Final:test
[INFO] |  +- org.apache.kafka:kafka-clients:jar:test:3.1.1:test
[INFO] |  +- org.apache.kafka:kafka-metadata:jar:3.1.1:test
[INFO] |  |  +- org.apache.kafka:kafka-server-common:jar:3.1.1:test
[INFO] |  |  +- org.apache.kafka:kafka-raft:jar:3.1.1:test
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.13.3:test
[INFO] |  |  \- com.yammer.metrics:metrics-core:jar:2.2.0:test
[INFO] |  +- org.apache.kafka:kafka-streams-test-utils:jar:3.1.1:test
[INFO] |  +- org.apache.kafka:kafka_2.13:jar:3.1.1:test
[INFO] |  |  +- org.scala-lang:scala-library:jar:2.13.6:test
[INFO] |  |  +- org.apache.kafka:kafka-storage:jar:3.1.1:test
[INFO] |  |  |  \- org.apache.kafka:kafka-storage-api:jar:3.1.1:test
[INFO] |  |  +- net.sourceforge.argparse4j:argparse4j:jar:0.7.0:test
[INFO] |  |  +- com.fasterxml.jackson.module:jackson-module-scala_2.13:jar:2.13.3:test
[INFO] |  |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.8:test
[INFO] |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.13.3:test
[INFO] |  |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:test
[INFO] |  |  +- org.bitbucket.b_c:jose4j:jar:0.7.8:test
[INFO] |  |  +- org.scala-lang.modules:scala-collection-compat_2.13:jar:2.4.4:test
[INFO] |  |  +- org.scala-lang.modules:scala-java8-compat_2.13:jar:1.0.0:test
[INFO] |  |  +- org.scala-lang:scala-reflect:jar:2.13.6:test
[INFO] |  |  +- com.typesafe.scala-logging:scala-logging_2.13:jar:3.9.3:test
[INFO] |  |  +- io.dropwizard.metrics:metrics-core:jar:4.2.11:test
[INFO] |  |  \- commons-cli:commons-cli:jar:1.4:test
[INFO] |  +- org.apache.kafka:kafka_2.13:jar:test:3.1.1:test
[INFO] |  \- org.junit.jupiter:junit-jupiter-api:jar:5.8.2:test
[INFO] |     +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] |     +- org.junit.platform:junit-platform-commons:jar:1.8.2:test
[INFO] |     \- org.apiguardian:apiguardian-api:jar:1.1.2:test
[INFO] \- org.apache.kafka:kafka-streams:jar:3.1.1:compile
[INFO]    +- org.rocksdb:rocksdbjni:jar:6.22.1.1:compile
[INFO]    +- org.slf4j:slf4j-api:jar:1.7.36:compile
[INFO]    +- com.fasterxml.jackson.core:jackson-annotations:jar:2.13.3:runtime
[INFO]    \- com.fasterxml.jackson.core:jackson-databind:jar:2.13.3:runtime
[INFO]       \- com.fasterxml.jackson.core:jackson-core:jar:2.13.3:runtime

All of the kafka versions are 3.1.1

MichaelZett commented 1 year ago

Hi Gary, thanks for looking into this. If one uses a spring-boot project, you are right: It will use the version from spring-boot. We have a project without spring-boot bom and try to comply to the versions manually. There we noticed, that spring-kafka-test 2.8.8 uses 3.0.1. Here is the gradle dependencyInsight:

org.apache.kafka:kafka_2.13:3.0.1 \--- org.springframework.kafka:spring-kafka-test:2.8.8 \--- testCompileClasspath So is our reasoning wrong, that all spring-projects in a given spring-boot version do use the versions listed for this spring-boot version.

MichaelZett commented 1 year ago

In your example, dependencyInsight looks like this: org.apache.kafka:kafka_2.13:3.0.1 -> 3.1.1 \--- org.springframework.kafka:spring-kafka-test:2.8.8 \--- testCompileClasspath (requested org.springframework.kafka:spring-kafka-test)

garyrussell commented 1 year ago

This is because 2.8.x is also used by Boot 2.6.x; which pulls in kafka 3.0.x. For that reason we cannot change 2.8.x to use the kafka 3.1.x natively.

You will need to specify the updated versions of all dependent jars if not using Boot's dependency management.

MichaelZett commented 1 year ago

Thanks for clarification. Will do.