openjdk-17 support - Githubissues

grunlab commented 9 months ago

Hi,

I'm currently building my joal container image based on a debian 11 image with openjdk-11 (https://gitlab.com/grunlab/joal/-/blob/main/build/Dockerfile?ref_type=heads)

I would like now to switch to debian 12 with openjdk-17 (openjdk-11 not available anymore) Do you plan to support openjdk-17 soon ?

Thank you

anthonyraymond commented 8 months ago

Have you encountered some errors with java 17? Usualy java is backward compatible and a program compiled for java 12 should run with 17 flawlessly

grunlab commented 8 months ago

Hi Anthony,

Yes, when I try to run Joal with openjdk-17 (https://gitlab.com/grunlab/joal/-/blob/main/build/Dockerfile_openjdk_17), I'm getting this error at container startup:

$ kc logs joal-6985cb7d6d-s2c7x -f
[INFO ] 2023-10-25 21:20:44.838 [           main] o.a.j.JackOfAllTradesApplication: Starting JackOfAllTradesApplication v2.1.34 using Java 17.0.8 on joal-6985cb7d6d-s2c7x with PID 8 (/opt/joal/jack-of-all-trades-2.1.34.jar started by k8s in /opt/joal)
[INFO ] 2023-10-25 21:20:44.943 [           main] o.a.j.JackOfAllTradesApplication: No active profile set, falling back to 1 default profile: "default"
[INFO ] 2023-10-25 21:21:14.020 [           main] o.s.b.w.e.t.TomcatWebServer: Tomcat initialized with port(s): 8080 (http)
[INFO ] 2023-10-25 21:21:14.443 [           main] o.a.c.h.Http11NioProtocol: Initializing ProtocolHandler ["http-nio-8080"]
[INFO ] 2023-10-25 21:21:14.518 [           main] o.a.c.c.StandardService: Starting service [Tomcat]
[INFO ] 2023-10-25 21:21:14.522 [           main] o.a.c.c.StandardEngine: Starting Servlet engine: [Apache Tomcat/9.0.65]
[INFO ] 2023-10-25 21:21:16.747 [           main] o.a.c.c.C.[.[.[/]: Initializing Spring embedded WebApplicationContext
[INFO ] 2023-10-25 21:21:16.818 [           main] o.s.b.w.s.c.ServletWebServerApplicationContext: Root WebApplicationContext: initialization completed in 30654 ms
[INFO ] 2023-10-25 21:21:42.446 [           main] o.s.s.w.DefaultSecurityFilterChain: Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@114f20c, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@1aa61f3, org.springframework.security.web.context.SecurityContextPersistenceFilter@169edd5, org.springframework.security.web.header.HeaderWriterFilter@13bb956, org.springframework.security.web.csrf.CsrfFilter@36bb2a, org.springframework.security.web.authentication.logout.LogoutFilter@1f21b6b, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@10332e1, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@356a0d, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@315f22, org.springframework.security.web.session.SessionManagementFilter@e949e1, org.springframework.security.web.access.ExceptionTranslationFilter@f9bbe, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@7e5441]
[INFO ] 2023-10-25 21:21:45.619 [           main] o.s.b.a.w.s.WelcomePageHandlerMapping: Adding welcome page: class path resource [public/index.html]
[INFO ] 2023-10-25 21:21:49.332 [           main] o.a.c.h.Http11NioProtocol: Starting ProtocolHandler ["http-nio-8080"]
[INFO ] 2023-10-25 21:21:49.825 [           main] o.s.b.w.e.t.TomcatWebServer: Tomcat started on port(s): 8080 (http) with context path ''
[INFO ] 2023-10-25 21:21:49.924 [           main] o.s.m.s.b.SimpleBrokerMessageHandler: Starting...
[INFO ] 2023-10-25 21:21:49.926 [           main] o.s.m.s.b.SimpleBrokerMessageHandler: BrokerAvailabilityEvent[available=true, SimpleBrokerMessageHandler [org.springframework.messaging.simp.broker.DefaultSubscriptionRegistry@1b865b5]]
[INFO ] 2023-10-25 21:21:49.932 [           main] o.s.m.s.b.SimpleBrokerMessageHandler: Started.
[INFO ] 2023-10-25 21:21:50.419 [           main] o.a.j.JackOfAllTradesApplication: Started JackOfAllTradesApplication in 73.313 seconds (JVM running for 93.495)
[INFO ] 2023-10-25 21:21:50.631 [           main] o.a.j.c.t.c.ConnectionHandler: Listening for incoming peer connections on port 49152
[INFO ] 2023-10-25 21:21:50.632 [           main] o.a.j.c.t.c.ConnectionHandler: Fetching ip from http://icanhazip.com/
[INFO ] 2023-10-25 21:21:50.926 [           main] o.a.j.c.t.c.ConnectionHandler: Successfully fetched public IP address: [82.64.9.53]
[INFO ] 2023-10-25 21:21:50.926 [           main] o.a.j.c.t.c.ConnectionHandler: IP reported to tracker will be: 82.64.9.53
[INFO ] 2023-10-25 21:21:53.425 [           main] o.a.j.c.c.JoalConfigProvider: App configuration has been successfully loaded
[ERROR] 2023-10-25 21:21:54.629 [           main] o.a.j.ApplicationReadyListener: Fatal error encountered
java.lang.IllegalStateException: Fatal error encountered
    at org.araymond.joal.ApplicationReadyListener.onApplicationEvent(ApplicationReadyListener.java:35)
    at org.araymond.joal.ApplicationReadyListener.onApplicationEvent(ApplicationReadyListener.java:16)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378)
    at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:114)
    at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:82)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:120)
    at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:114)
    at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:82)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:323)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295)
    at org.araymond.joal.JackOfAllTradesApplication.main(JackOfAllTradesApplication.java:15)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: java.io.UncheckedIOException: java.nio.file.FileSystemException: /data/clients: Value too large for defined data type
    at java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
    at java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
    at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
    at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
    at org.araymond.joal.core.client.emulated.BitTorrentClientProvider.listClientFiles(BitTorrentClientProvider.java:53)
    at org.araymond.joal.core.SeedManager.listClientFiles(SeedManager.java:171)
    at org.araymond.joal.core.SeedManager.startSeeding(SeedManager.java:122)
    at org.araymond.joal.ApplicationReadyListener.onApplicationEvent(ApplicationReadyListener.java:33)
    ... 24 more
Caused by: java.nio.file.FileSystemException: /data/clients: Value too large for defined data type
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
    at java.base/sun.nio.fs.UnixException.asIOException(UnixException.java:115)
    at java.base/sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.readNextEntry(UnixDirectoryStream.java:168)
    at java.base/sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.hasNext(UnixDirectoryStream.java:198)
    at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:349)
    at java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:83)
    ... 36 more
[INFO ] 2023-10-25 21:21:54.731 [           main] o.a.j.ApplicationClosingListener: Gracefully shutting down application...
[INFO ] 2023-10-25 21:21:54.733 [       Thread-0] o.a.j.c.t.c.ConnectionHandler: IP fetcher thread has been stopped
[INFO ] 2023-10-25 21:21:54.822 [           main] o.a.j.ApplicationClosingListener: JOAL gracefully shut down
[INFO ] 2023-10-25 21:21:56.436 [           main] o.a.c.h.Http11NioProtocol: Pausing ProtocolHandler ["http-nio-8080"]
[INFO ] 2023-10-25 21:21:56.520 [           main] o.a.c.c.StandardService: Stopping service [Tomcat]
[INFO ] 2023-10-25 21:21:56.721 [           main] o.a.c.h.Http11NioProtocol: Stopping ProtocolHandler ["http-nio-8080"]
[ERROR] 2023-10-25 21:21:56.824 [http-nio-8080-exec-2] o.a.c.h.Http11Processor: Error processing request
java.lang.NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "this.text" is null
    at java.base/java.util.regex.Matcher.getTextLength(Matcher.java:1769)
    at java.base/java.util.regex.Matcher.reset(Matcher.java:415)
    at java.base/java.util.regex.Matcher.<init>(Matcher.java:252)
    at java.base/java.util.regex.Pattern.matcher(Pattern.java:1134)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:624)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:833)
[ERROR] 2023-10-25 21:21:56.824 [http-nio-8080-exec-1] o.a.c.h.Http11Processor: Error processing request
java.lang.NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "this.text" is null
    at java.base/java.util.regex.Matcher.getTextLength(Matcher.java:1769)
    at java.base/java.util.regex.Matcher.reset(Matcher.java:415)
    at java.base/java.util.regex.Matcher.<init>(Matcher.java:252)
    at java.base/java.util.regex.Pattern.matcher(Pattern.java:1134)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:624)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:833)
[INFO ] 2023-10-25 21:21:56.922 [           main] o.a.c.h.Http11NioProtocol: Destroying ProtocolHandler ["http-nio-8080"]

No issue using the same Dockerfile with debian:11 as based image and openjdk-11 installed (https://gitlab.com/grunlab/joal/-/blob/main/build/Dockerfile)

Thank you

anthonyraymond commented 8 months ago

I see, i'll take a look at that ASAP. Why are you packaging JOAL on your own though? Is there something missing from the official JOAL docker image?

grunlab commented 6 months ago

Hi Anthony,

Sorry my late answer !

I've made my own joal image for those following reasons:

As far as I can, I prefer to run my projects using my own custom base-image for which I know exactly which packages are present or not (for example: to have "kubectl cp" working, tar must be installed, etc etc ...)
Joal official image is running by default as root. I prefer to run my projects as non-root (in my case: with the user k8s : uid:1000/gid:1000)
I'm running Joal on a kubernetes cluster with the joal-conf part stored on a persistentVolume (which is by default empty and then preventing Joal to run at the first start-up). So I've added into my custom entrypoint a copy of the needed data-structure (config.json, clients and torrents) to initialize the persisentVolume at the first run.

Did you progress for openjdk-17 support ? (still KO with Joal 2.1.36)

Thank you

grunlab commented 2 months ago

Hi Anthony,

Any progress ? Debian 11 Security Support end in 3 months (31 Jul 2024) Would like to switch to Debian 12 / OpenJDK-17 asap.

Thank you

anthonyraymond commented 2 months ago

It works with Java17 on my environment.

The error does not seems related to java: /data/clients: Value too large for defined data type.

Any chances you are using a 32b version of java?

grunlab commented 2 months ago

Yes, I confirm, I'm building the image to run on arm v7 arch (so 32 bit)

anthonyraymond commented 2 months ago

Can you share the client file you are using please? The client.json file will reference a .client file. Can you please upload this .client file here?

anthonyraymond commented 2 months ago

Also, could you try to build the .jar artifact from a multistage image using the proper arch (like in m'y DockerFile?) https://github.com/anthonyraymond/joal/blob/master/Dockerfile

grunlab commented 2 months ago

Can you share the client file you are using please? The client.json file will reference a .client file. Can you please upload this .client file here?

Not sure to understand which client.json / .client files you are talking about. I suppose it is the client specified into my config.json file --> deluge-2.1.1.client

$ cat config.json 
{
  "minUploadRate" : 30,
  "maxUploadRate" : 170,
  "simultaneousSeed" : 5,
  "client" : "deluge-2.1.1.client",
  "keepTorrentWithZeroLeechers" : true,
  "uploadRatioTarget" : -1.0
}

$ cat clients/deluge-2.1.1.client 
{
    "keyGenerator": {
        "algorithm": {
            "type": "HASH_NO_LEADING_ZERO",
            "length": 8
        },
        "refreshOn": "TORRENT_PERSISTENT",
        "keyCase": "upper"
    },
    "peerIdGenerator": {
        "algorithm": {
            "type": "REGEX",
            "pattern": "-DE211s-[A-Za-z0-9_~\\(\\)\\!\\.\\*-]{12}"
        },
        "refreshOn": "NEVER",
        "shouldUrlEncode": false
    },
    "urlEncoder": {
        "encodingExclusionPattern": "[A-Za-z0-9_~\\(\\)\\!\\.\\*-]",
        "encodedHexCase": "lower"
    },
    "query": "info_hash={infohash}&peer_id={peerid}&port={port}&uploaded={uploaded}&downloaded={downloaded}&left={left}&corrupt=0&key={key}&event={event}&numwant={numwant}&compact=1&no_peer_id=1&supportcrypto=1&redundant=0",
    "numwant": 200,
    "numwantOnStop": 0,
    "requestHeaders": [
        { "name": "User-Agent", "value": "Deluge/2.1.1 libtorrent/1.2.15.0" },
        { "name": "Accept-Encoding", "value": "gzip" },
        { "name": "Connection", "value": "close" }
    ]
}

FYI, I've tried with other clients : same issue.

Also, could you try to build the .jar artifact from a multistage image using the proper arch (like in m'y DockerFile?) https://github.com/anthonyraymond/joal/blob/master/Dockerfile

OK done:

Dockerfile: https://gitlab.com/grunlab/joal/-/blob/main/build/Dockerfile
Build logs: https://gitlab.com/grunlab/joal/-/jobs/6743996552
Image: registry.gitlab.com/grunlab/joal:2.1.36-multistage-arm-v7

I'm getting exactly the same error when running this image.

anthonyraymond commented 2 months ago

i can't understand why this would be related to a JVM version... I truely believe the problem is somewhere else. But i have no idea where to look for...

Can you try with debian 12 and openjdk 11 (if available from repo)?

Also can you run the following commands from within the container (before it crash) :

ls -la /data
ls -la /data/clients

And starting to mess around with weird idea... Can you try :

with openjdk-17-jre instead of openjdk-17-jre-headless
without the --no-install-recommends flag

grunlab commented 2 months ago

And starting to mess around with weird idea... Can you try :

with openjdk-17-jre instead of openjdk-17-jre-headless

without the --no-install-recommends flag

I've tried but this did not solve the issue.

i can't understand why this would be related to a JVM version... I truely believe the problem is somewhere else. But i have no idea where to look for...

You were right and I think I finally found the root cause of the issue.

Until now, I was using a cephfs volume for the persistence of /data --> https://gitlab.com/grunlab/joal/-/blob/main/deploy/joal.yml#L73-L82

Removing the persistence or replacing the cephfs volume by an NFS one did "solved" the issue --> https://gitlab.com/grunlab/joal/-/blob/main/deploy/joal_workaround_with_nfs.yml#L83-L86

With this "workaround" configuration, I'm now able to run joal properlly on debian_12 / openjdk_17 without any error.

So conclusion, java 17 did break somewhere the "compatibility" with cephfs usage for persistence ... Not sure it is something you can fix/mitigate at application level !?

anthonyraymond commented 2 months ago

Indeed, i don't think i can do anything at the application level. It's ether an OS or JVM issue

anthonyraymond / joal

openjdk-17 support #218