jetty application with jersey above 3.1.4 complains wrong java version and crashes

jarinek commented 4 days ago

I have an application with the following dependencies

    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-server</artifactId>
        <version>12.0.15</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-util</artifactId>
        <version>12.0.15</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty.ee10</groupId>
        <artifactId>jetty-ee10-servlet</artifactId>
        <version>12.0.15</version>
    </dependency>
    <dependency>
        <groupId>org.glassfish.jersey.core</groupId>
        <artifactId>jersey-server</artifactId>
        <version>3.1.4</version>
    </dependency>
    <dependency>
        <groupId>org.glassfish.jersey.containers</groupId>
        <artifactId>jersey-container-jetty-http</artifactId>
        <version>3.1.4</version>
    </dependency>
    <dependency>
        <groupId>org.glassfish.jersey.containers</groupId>
        <artifactId>jersey-container-servlet-core</artifactId>
        <version>3.1.4</version>
    </dependency>
    <dependency>
        <groupId>org.glassfish.jersey.inject</groupId>
        <artifactId>jersey-hk2</artifactId>
        <version>3.1.4<</version>
    </dependency>
    <dependency>
        <groupId>org.glassfish.jersey.core</groupId>
        <artifactId>jersey-client</artifactId>
        <version>3.1.4<</version>
    </dependency>

The application positively runs with java 21 but crashes with the following error message

2024-11-08 11:44:13.995 INFO [main] Launcher - java.specification.version:21 2024-11-08 11:44:14.195 INFO [main] Launcher - sun.jnu.encoding:UTF-8 2024-11-08 11:44:14.195 INFO [main] Launcher - java.class.path:/app/kds-3.0.jar 2024-11-08 11:44:14.195 INFO [main] Launcher - java.vm.vendor:Azul Systems, Inc. .. .. .. 2024-11-08 11:44:32.342 ERROR [main] Launcher - Unable to run embedded application server: jakarta.ws.rs.ProcessingException: Jetty container is not supported on JDK version less than 17. at org.glassfish.jersey.jetty.JettyHttpContainerFactory.createServer(JettyHttpContainerFactory.java:227) ~[kds-3.0.jar:?] at org.glassfish.jersey.jetty.JettyHttpContainerFactory.createServer(JettyHttpContainerFactory.java:73) ~[kds-3.0.jar:?] at de.telekom.bdmp.kds.Launcher.main(Launcher.java:80) [kds-3.0.jar:?] 2024-11-08 11:44:32.440 INFO [main] KeyService - Canceling internal Timer thread 2024-11-08 11:44:32.440 INFO [main] KeyService - Internal Timer thread cancelled

The test has been run for all version 3.1.4 till 3.1.9 with the same result. The highest jersey version which works is 3.1.3

senivam commented 4 days ago

the very first dependency in your list is

<dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-server</artifactId>
        <version>12.0.15</version>
    </dependency>

it does not work on JDKs prior to 17. Jersey 3.1.3 and earlier uses Jetty 11 (which works on JDKs lesser than 17) so it does not refer to the dependency you are listing. Jersey 3.1.4 starts to use Jetty 12 which already takes the dependency you are using (12.0.15). So, either stop using incompatible dependencies or use proper JDK.

jarinek commented 4 days ago

I am not sure if you have read my post. I use java 21. What do you mean by "stop using incompatible dependencies"? Could you please point to some documentation which explains the relations?

senivam commented 4 days ago

I was presuming, that the sentence

The application positively runs with java 21 but crashes with the following error message

means that it works on JDK 21 but fails with some other JDK. If it all refers to the JDK 21, all dependencies are compatible with the used JDK.

However, to investigate this further, it would be nice to have some glue on what exactly fails. So, could you please provide some reproducer?

senivam commented 4 days ago

regarding documentation - a word about changes in Jetty modules can be found here

jarinek commented 3 days ago

3 statements

My application is compiled and runs in java 21.
It works with jersey 3.1.3.
It crashes with jersey 3.1.4+ claiming that the java version in use is less than 17. (That is weird, isn't it?)

It might be that the dependencies and the application code are somehow "illegal", but on the other hand, all dependencies are all up-to-date (except for jersey) and all claiming to be compatible with java 21.

This is the java version: openjdk version "21.0.5" 2024-10-15 LTS OpenJDK Runtime Environment Zulu21.38+21-CA (build 21.0.5+11-LTS) OpenJDK 64-Bit Server VM Zulu21.38+21-CA (build 21.0.5+11-LTS, mixed mode, sharing)

Concerning reproducer, I will try ...

senivam commented 1 day ago

basing on your report I've created a simple hello world project with given dependencies and indeed I can confirm that just out of the box it fails while using Jersey 3.1.4, Jetty 12, and JDK 21.

I also recall that we had an issue with that release of Jersey (3.1.4) because the release did not respect the proper structure of the Multi-relese jar for the Jetty artifacts.

Then I was checking how other versions of Jersey and Jetty 12 start on JDK 21 and for versions since the 3.1.6 till the latest (3.1.9) everything starts fine. So, I enclose the list of my dependencies:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.glassfish.jersey.issues</groupId>
    <artifactId>issue_5793</artifactId>
    <version>1.0-SNAPSHOT</version>
    <name>issue_5793</name>
    <packaging>war</packaging>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <junit.version>5.8.1</junit.version>
        <jersey.version>3.1.6</jersey.version>
        <jetty.version>12.0.15</jetty.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-server</artifactId>
            <version>${jetty.version}</version>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-util</artifactId>
            <version>${jetty.version}</version>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty.ee10</groupId>
            <artifactId>jetty-ee10-servlet</artifactId>
            <version>${jetty.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.core</groupId>
            <artifactId>jersey-server</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.containers</groupId>
            <artifactId>jersey-container-jetty-http</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.containers</groupId>
            <artifactId>jersey-container-servlet-core</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.inject</groupId>
            <artifactId>jersey-hk2</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.core</groupId>
            <artifactId>jersey-client</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.11.3</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.test-framework</groupId>
            <artifactId>jersey-test-framework-core</artifactId>
            <scope>test</scope>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.test-framework.providers</groupId>
            <artifactId>jersey-test-framework-provider-jetty</artifactId>
            <version>${jersey.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.connectors</groupId>
            <artifactId>jersey-jetty-connector</artifactId>
            <version>${jersey.version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <version>3.3.2</version>
            </plugin>
        </plugins>
    </build>
</project>

jarinek commented 1 day ago

Hi, interessting, so far, I was not able to simplify my application to and reproduce the failure. In my case, the application really fails with all jersey version starting with 3.1.4 till 3.1.9. and starts to work again with 4.0.0-M1

There is also similar issue https://github.com/eclipse-ee4j/jersey/issues/5694

kind of unresolved, IMHO. The issues is obviously not same, possibly there are multiple situations, where the failure can occur.

I also have to note, that I use maven-assembly-plugin to create a single application jar, and have some suspect, that this might bring some clashes.

senivam commented 1 day ago

Hi @jarinek, I have some questions:

do you use Jersey from MVN central or from some custom build?
could you try to clean the local .m2 repo (enough to remove org.glassfish.jersey folder) for the Jersey to be re-downloaded?
can you try to run the app on some other JDK than from Azul Systems, Inc.?
do you run the app in some OSGi environment (like Felix)?

regarding the issue you are mentioning (#5694), there appeared that Jersey was customly built and had wrong OSGi headers in the MANIFEST.MF that is why the issue was closed without visible resolution. The reporter added this information by e-mail communication.

senivam commented 18 hours ago

I've just tried my reproducer with JDK 21 from Azul Systems:

Maven home: /media/storage/software/maven/apache-maven-3.9.9
Java version: 21.0.5, vendor: Azul Systems, Inc., runtime: /media/storage/software/java/zulu21.38.21-ca-jdk21.0.5-linux_x64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.8.0-48-generic", arch: "amd64", family: "unix"

and it behaves similarly to Open JDK from Oracle: fails for Jersey 3.1.4 (because of the mentioned issue with Multi-Release jar) and runs for versions [3.1.6 : 3.1.9]

jarinek commented 17 hours ago

In my case, the dependencies are fetched from a corporate mirror of maven central. There is for sure no custom build in place. The build process runs in gitlab in a k8s runner, i.e., all is fetched and build from scratch. I probably cannot run the application on another JRE due corporate policie. The JRE image is provided and updated centrally. The app docker image is build up on it. Pretty controlled environment. I have tried to analyze the dependencies and played around it, but without success. I also have difficulties to get any log message or more detail from jersey, although I have added org.slf4j:jul-to-slf4j:jar:2.0.16. So, I strugle to separate the issue cause. To create a demo will really take time, the application have some external complex dependecies, which are hard to mock.

jansupol commented 16 hours ago

@jarinek can you check your .m2 version of the jersey-container-jetty-http jar and see if it contains META-INF/versions/17 folder with java classes?

A workaround is to use jetty11 compatible jersey-container-jetty11-http that has been used with Jersey 3.1.3.

senivam commented 16 hours ago

So, I think you can publish somewhere the jersey-container-jetty-http-<VERSION>.jar which you are using so we can compare it with that from mvn central. I'm assuming your company's policies can not deny this because Jersey is open source.

and also could you post the output of the mvn dependency:tree command for your project?

jarinek commented 16 hours ago

The result of tree and the analyse maven tasks (the later results confuses me)

34310 [INFO] [1m--- [0;32mdependency:3.8.1:analyze[m [1m(default-cli)[m @ [36mkds[0;1m ---[m 41662 [WARNING] Used undeclared dependencies found: 41662 [WARNING] jakarta.inject:jakarta.inject-api:jar:2.0.1:compile 41662 [WARNING] jakarta.ws.rs:jakarta.ws.rs-api:jar:3.1.0:compile 41662 [WARNING] jakarta.servlet:jakarta.servlet-api:jar:6.0.0:compile 41662 [WARNING] org.glassfish.hk2:hk2-api:jar:3.0.6:compile 41662 [WARNING] jakarta.annotation:jakarta.annotation-api:jar:2.1.1:compile 41662 [WARNING] Unused declared dependencies found: 41662 [WARNING] org.glassfish.jersey.inject:jersey-hk2:jar:3.1.9:compile 41662 [WARNING] org.apache.logging.log4j:log4j-api:jar:2.24.1:runtime 41662 [WARNING] org.apache.logging.log4j:log4j-core:jar:2.24.1:runtime 41662 [WARNING] org.apache.logging.log4j:log4j-slf4j2-impl:jar:2.24.1:runtime 41662 [WARNING] org.slf4j:jul-to-slf4j:jar:2.0.16:runtime 41662 [WARNING] jakarta.activation:jakarta.activation-api:jar:2.1.3:runtime 41662 [INFO] 41662 [INFO] [1m--- [0;32mdependency:3.8.1:treeINFO[m @ [36mkds[0;1m ---[m 41672 [INFO] de.telekom.ail:kds:jar:3.0 41672 [INFO] +- org.eclipse.jetty:jetty-server:jar:12.0.15:compile 41672 [INFO] | - org.eclipse.jetty:jetty-http:jar:12.0.15:compile 41672 [INFO] +- org.eclipse.jetty:jetty-util:jar:12.0.15:compile 41672 [INFO] +- org.eclipse.jetty.ee10:jetty-ee10-servlet:jar:12.0.15:compile 41672 [INFO] | +- jakarta.servlet:jakarta.servlet-api:jar:6.0.0:compile 41672 [INFO] | +- org.eclipse.jetty:jetty-security:jar:12.0.15:compile 41672 [INFO] | - org.eclipse.jetty:jetty-session:jar:12.0.15:compile 41672 [INFO] +- org.eclipse.jetty:jetty-io:jar:12.0.15:compile 41672 [INFO] +- org.glassfish.jersey.core:jersey-common:jar:3.1.9:compile 41672 [INFO] | +- jakarta.ws.rs:jakarta.ws.rs-api:jar:3.1.0:compile 41672 [INFO] | +- jakarta.annotation:jakarta.annotation-api:jar:2.1.1:compile 41672 [INFO] | +- jakarta.inject:jakarta.inject-api:jar:2.0.1:compile 41672 [INFO] | - org.glassfish.hk2:osgi-resource-locator:jar:1.0.3:compile 41672 [INFO] +- org.glassfish.jersey.core:jersey-server:jar:3.1.9:compile 41672 [INFO] | +- org.glassfish.jersey.core:jersey-client:jar:3.1.9:compile 41672 [INFO] | - jakarta.validation:jakarta.validation-api:jar:3.0.2:compile 41672 [INFO] +- org.glassfish.jersey.containers:jersey-container-jetty-http:jar:3.1.9:compile 41672 [INFO] +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:3.1.9:compile 41672 [INFO] +- org.glassfish.jersey.inject:jersey-hk2:jar:3.1.9:compile 41672 [INFO] | +- org.glassfish.hk2:hk2-locator:jar:3.0.6:compile 41672 [INFO] | | +- org.glassfish.hk2.external:aopalliance-repackaged:jar:3.0.6:compile 41672 [INFO] | | +- org.glassfish.hk2:hk2-api:jar:3.0.6:compile 41672 [INFO] | | - org.glassfish.hk2:hk2-utils:jar:3.0.6:compile 41672 [INFO] | - org.javassist:javassist:jar:3.30.2-GA:compile 41672 [INFO] +- commons-cli:commons-cli:jar:1.9.0:compile 41672 [INFO] +- commons-io:commons-io:jar:2.17.0:compile 41672 [INFO] +- org.ini4j:ini4j:jar:0.5.4:compile 41673 [INFO] +- org.json:json:jar:20240303:compile 41673 [INFO] +- org.apache.logging.log4j:log4j-api:jar:2.24.1:runtime 41673 [INFO] +- org.apache.logging.log4j:log4j-core:jar:2.24.1:runtime 41673 [INFO] +- org.apache.logging.log4j:log4j-slf4j2-impl:jar:2.24.1:runtime 41673 [INFO] +- org.slf4j:jul-to-slf4j:jar:2.0.16:runtime 41673 [INFO] +- org.slf4j:slf4j-api:jar:2.0.16:compile 41673 [INFO] +- com.google.code.gson:gson:jar:2.11.0:compile 41673 [INFO] | - com.google.errorprone:error_prone_annotations:jar:2.27.0:compile 41673 [INFO] +- io.gsonfire:gson-fire:jar:1.9.0:compile 41673 [INFO] +- com.squareup.okio:okio-jvm:jar:3.9.1:compile 41673 [INFO] | - org.jetbrains.kotlin:kotlin-stdlib:jar:1.9.25:compile 41673 [INFO] | - org.jetbrains:annotations:jar:13.0:compile 41673 [INFO] +- com.auth0:java-jwt:jar:4.4.0:compile 41673 [INFO] | - com.fasterxml.jackson.core:jackson-databind:jar:2.14.2:runtime 41673 [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.14.2:runtime 41673 [INFO] | - com.fasterxml.jackson.core:jackson-core:jar:2.14.2:runtime 41673 [INFO] +- jakarta.activation:jakarta.activation-api:jar:2.1.3:runtime 41673 [INFO] - org.projectlombok:lombok:jar:1.18.34:provided

jarinek commented 15 hours ago

Maybe this might help instead of upload (it a fetch from the mirror on my desktop)

$ cat jersey-container-jetty-http-3.1.9.jar.sha1 44fa54432c4c16f19d5f05923707ccb3dd8f65a4

$ cksum jersey-container-jetty-http-3.1.9.jar 448573842 56702 jersey-container-jetty-http-3.1.9.jar

jarinek commented 15 hours ago

@jarinek can you check your .m2 version of the jersey-container-jetty-http jar and see if it contains META-INF/versions/17 folder with java classes?

A workaround is to use jetty11 compatible jersey-container-jetty11-http that has been used with Jersey 3.1.3.

Actually, the application does work with jersey 3.1.3 (and I if am not mistaken also with jetty11). Thus it is not a fundamental blocker.

senivam commented 15 hours ago

regarding checksums and SHA1 hash - all correspond to those in the mvn central.

eclipse-ee4j / jersey

jetty application with jersey above 3.1.4 complains wrong java version and crashes #5793