Incompatibility with spring-kafka-test

DarkAtra commented 7 months ago

The latest version of this starter is incompatible with spring-kafka-test as it expects a different version of apache commons.

Environment:

Github Actions (ubuntu-latest)
Github Codespaces (Ubuntu Linux)

Steps to reproduce:

checkout the main branch of this repo: https://github.com/DarkAtra/flapdoodle-spring-kafka-issue
confirm that your system has at least java 21 installed
run the tests ./mvnw clean verify

The error message:

Description:

An attempt was made to call a method that does not exist. The attempt was made from the following location:

    org.apache.commons.compress.utils.IOUtils.skip(IOUtils.java:282)

The following method did not exist:

    'long org.apache.commons.io.IOUtils.skip(java.io.InputStream, long, java.util.function.Supplier)'

The calling method's class, org.apache.commons.compress.utils.IOUtils, was loaded from the following location:

    jar:file:/home/codespace/.m2/repository/org/apache/commons/commons-compress/1.26.1/commons-compress-1.26.1.jar!/org/apache/commons/compress/utils/IOUtils.class

The called method's class, org.apache.commons.io.IOUtils, is available from the following locations:

    jar:file:/home/codespace/.m2/repository/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar!/org/apache/commons/io/IOUtils.class

The called method's class hierarchy was loaded from the following locations:

    org.apache.commons.io.IOUtils: file:/home/codespace/.m2/repository/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar

Action:

Correct the classpath of your application so that it contains compatible versions of the classes org.apache.commons.compress.utils.IOUtils and org.apache.commons.io.IOUtils

Running ./mvnw clean dependency:tree reveals that:

spring-kafka-test pulls commons-io:commons-io:jar:2.11.0
flapdoodle pulls org.apache.commons:commons-compress:jar:1.26.1 which apparently is bytecode-incompatible with commons-io 2.11.0

As a user of both spring kafka and flapdoodle, i'd expect flapdoodle to play nicely with all spring libraries.

michaelmosmann commented 7 months ago

@DarkAtra ... i guess that i can fix that..

michaelmosmann commented 6 months ago

@DarkAtra .. i guess i can not fix this.. yes.. i could use an older version of commons-compress which matches your commons-io version, but as this is not recommended, i would suggest that you enforce an older version of common-compress:

    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-compress</artifactId>
        <version>1.23.0</version>
    </dependency>

.. should do the trick, as these version uses commons-io:2.11.0 .. and works with flapdoodle. Maybe you should ask the spring-kafka-team to update their dependency..

DarkAtra commented 6 months ago

@DarkAtra .. i guess i can not fix this.. yes.. i could use an older version of commons-compress which matches your commons-io version, but as this is not recommended, i would suggest that you enforce an older version of common-compress:
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-compress</artifactId>
        <version>1.23.0</version>
    </dependency>
.. should do the trick, as these version uses commons-io:2.11.0 .. and works with flapdoodle. Maybe you should ask the spring-kafka-team to update their dependency..

There's also other libraries which require an older version of apache commons, for example the apache batik project (which we also use). Also, i'd personally expect any spring boot starter to be fully compatible with the corresponding spring boot version. An alternative would be to shade dependencies of flapdoodle so that conflicts are no longer an issue.

We resorted to rollback to an older flapdoodle version for our projects as a workaround.

michaelmosmann commented 6 months ago

@DarkAtra .. i understand. What was the error message if you force this common-compress version (or an older one)?

DarkAtra commented 6 months ago

@DarkAtra .. i understand. What was the error message if you force this common-compress version (or an older one)?

Oh, i haven't tried pinning the common-compress version. I think from our point of view it makes more sense to pin the flapdoodle version instead since that's what we rely on directly. And the commona-compress part is just an implementation detail.

I guess your suggested fix would work as well though.

michaelmosmann commented 6 months ago

@DarkAtra .. i tried to reproduce this error.. but it works on my machine (https://github.com/DarkAtra/flapdoodle-spring-kafka-issue, ubuntu 22.xx)

DarkAtra commented 6 months ago

@DarkAtra .. i tried to reproduce this error.. but it works on my machine (https://github.com/DarkAtra/flapdoodle-spring-kafka-issue, ubuntu 22.xx)

We ran our tests in Github Actions and on Github Codespaces and in both situations they did not pass but threw the above error. However, we were unable to reproduce it locally on our development machine (M2 Macbook).

I've added the corresponding codespace configuration to the reproducer, you should be able to create a codespace after forking the repo and then just run the tests with ./mvnw clean verify to reproduce.

Regarding the depdency tree thingy: yes, commons-compress does not pull commons-io but it still uses it internally for some reason.

org.apache.commons.compress.utils.IOUtils:

michaelmosmann commented 6 months ago

@DarkAtra .. ah.. got it. If the mongo executable could be extracted one, common compress will not be called again. So thats why the bug hides sometimes.. i tried to set commons-compress to 1.23.0 .. so it works in this case.

commons-compress depends on commons-io, but maven does not show it clearly...

michaelmosmann commented 6 months ago

@DarkAtra .. i will see if i can find an alternative solution for common-compress.. but it will take time.. i do not have any experience with shadowing dependencies..

DarkAtra commented 6 months ago

@DarkAtra .. ah.. got it. If the mongo executable could be extracted one, common compress will not be called again. So thats why the bug hides sometimes.. i tried to set commons-compress to 1.23.0 .. so it works in this case.

commons-compress depends on commons-io, but maven does not show it clearly...

i see, that explains why we couldn't reproduce it on our local dev machines.

@DarkAtra .. i will see if i can find an alternative solution for common-compress.. but it will take time.. i do not have any experience with shadowing dependencies..

thanks, we'll stick to the workaround in the meantime

DarkAtra commented 2 months ago

hey @michaelmosmann, i just wanted to check if there has been any progress on this one. We just tried updating flapdoodle to the latest version and ran into the same issue again (with spring boot 3.2.8).

michaelmosmann commented 2 months ago

@DarkAtra .. sorry. no progress on this one..

flapdoodle-oss / de.flapdoodle.embed.mongo.spring

Incompatibility with spring-kafka-test #50