Class Not Found: Threadpool

[shadycuz]

The example docker-simple example no longer works.

Something either changed in a Jenkins or JFR.

I'm not sure where to even begin =/

Releated #656

Error

docker run --rm -v $(pwd):/workspace jfr-test
java.lang.RuntimeException: Unhandled exception
        at io.jenkins.jenkinsfile.runner.bootstrap.commands.JenkinsLauncherCommand.call(JenkinsLauncherCommand.java:69)
        at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.call(Bootstrap.java:71)
        at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.call(Bootstrap.java:21)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
        at picocli.CommandLine.execute(CommandLine.java:2078)
        at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.main(Bootstrap.java:46)
Caused by: java.lang.NoClassDefFoundError: org/eclipse/jetty/util/thread/ThreadPool
        at io.jenkins.jenkinsfile.runner.App.run(App.java:25)
        at io.jenkins.jenkinsfile.runner.bootstrap.commands.JenkinsLauncherCommand.runJenkinsfileRunnerApp(JenkinsLauncherCommand.java:226)
        at io.jenkins.jenkinsfile.runner.bootstrap.commands.JenkinsLauncherCommand.call(JenkinsLauncherCommand.java:67)
        ... 10 more
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.util.thread.ThreadPool
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 13 more

[oleg-nenashev]

It is a valid bug. TL;DR: never use "latest" in Docker :D

[shadycuz]

@oleg-nenashev I'm currently trying to narrow down the version of Jenkins, but installing plugins is crazy slow.

jenkins-plugin-cli --plugins workflow-aggregator

Noticed this yesterday as well, was getting timeout errors or something (don't remember). I thought it might be a hickup in the internet but maybe its something else going on with Jenkins infra?

I'm pretty sure this worked in march but not april.

[shadycuz]

tag 2.319.3-lts-jdk11 works, trying 2.332.1-lts-jdk11 now.

I'm just trying all the minor releases until I found the one that broke things.

Edit: 2.332.1 worked trying 2.346.1 now.

Edit: 2.346.1 works, trying 2.361.1

This is slow because of these errors that I now see in 2.361.1 when installing plugins.

Nov 03, 2022 9:19:31 AM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://updates.jenkins.io:443: The target server failed to respond
Nov 03, 2022 9:19:31 AM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://updates.jenkins.io:443

Edit: It looks like all the LTS releases are safe? The latest-jdk that seems to break things is the next jenkins version 2.376? Does that mean possible breaking changes for JFR?

[shadycuz]

@oleg-nenashev I finally have it narrowed down. It actually broke pretty recently, my personal pipeline must be failing for other reasons.

This combination works:

ARG baseImage=jenkins/jenkins:2.346.3-lts-jdk11

# Load the JFR docker image
FROM jenkins/jenkinsfile-runner:latest as jfr

Everything from v2.361.1 and onwards doesn't work. There is a couple of things in the change log that look like it could be the reason but I'm not sure which one it is.

This looks suspicious. I will try it first:

Use native Java Platform functionality rather than Ant to load classes. The old behavior can be restored by setting -Dhudson.ClassicPluginStrategy.useAntClassLoader=true. (pull 6571)

EDIT: Settings

ENV JAVA_OPTS -Dhudson.ClassicPluginStrategy.useAntClassLoader=true

Did not fix the error

[shadycuz]

Actually on second thought, I'm not sure if setting the `AVA_OPTS is the correct way to try and fix this in JFR? Mayne I need to do it differently?

[oleg-nenashev]

JAVA_OPTS should work in JFR. I am not sure what breaks TBH, needs to test it on my own

[IreshMM]

Where can I find Jenkins Docker image source releases? That may help to pinpoint the change that causes issue. https://github.com/jenkinsci/docker/tags doesn't seem to have any recent releases.

[IreshMM]

Some digging into the Docker images revealed the cause.

After version 2.357-jdk11 (tag) winstore.jar has been moved to executable/winstore.jar inside Jenkins war file

Following can be a temporary fix,

RUN ln -s /usr/share/jenkins/executable/winstone.jar /usr/share/jenkins/winstone.jar

[shadycuz]

@IreshMM I have yet to test this but what you said makes sense. Good job spotting that, I looked at the Jenkins change log for hours and did narrow it down to version 2.357 but not the "why" part.

@oleg-nenashev I'm not sure what needs fixed in the runner but it should be pretty simple? @IreshMM maybe you would want to submit a PR bugfix?

EDIT: I just searched for winstore.jar in this repo and I did not find that hardcoded path... so maybe its a more difficult fix?

[oleg-nenashev]

i will be timnkering with the repository in the coming weeks, so I hope to restore all demos or remove them

[lhupfeldt]

The Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.util.thread.ThreadPool is not specifically related to the docker image, it also happens when using jenkins.war

Maybe change the issue title?

Tried a few combinations, including:

]$ java --version
openjdk 11.0.18 2023-01-17
OpenJDK Runtime Environment (Red_Hat-11.0.18.0.10-1.fc37) (build 11.0.18+10)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.18.0.10-1.fc37) (build 11.0.18+10, mixed mode, sharing)

]$ java --version
openjdk 17.0.6 2023-01-17
OpenJDK Runtime Environment (Red_Hat-17.0.6.0.10-1.fc37) (build 17.0.6+10)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.6.0.10-1.fc37) (build 17.0.6+10, mixed mode, sharing)

JENKINS_VERSION = "2.375.3"
PLUGIN_MANAGER_VERSION = "2.12.11"
JENKINSFILE_RUNNER_VERSION = "1.0-beta-31"

[akoerner1]

i will be timnkering with the repository in the coming weeks, so I hope to restore all demos or remove them

please dont remove the docker demo.

[shadycuz]

@oleg-nenashev any ideas?