jenkinsci / jenkinsfile-runner

A command line tool to run Jenkinsfile as a function
MIT License
1.13k stars 291 forks source link

Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.util.thread.ThreadPool #656

Closed shadycuz closed 1 year ago

shadycuz commented 1 year ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.375 OS: Linux - 5.10.102.1-microsoft-standard-WSL2 --- ace-editor:1.1 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 caffeine-api:2.9.3-65.v6a_47d0f4d1fe cloudbees-folder:6.773.vd2dcc704ee7e command-launcher:1.2 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-27.vb_fa_3896786a_7 credentials:1189.vf61b_a_5e2f62e credentials-binding:523.vd859a_4b_122e6 display-url-api:2.3.6 durable-task:501.ve5d4fc08b0be filesystem_scm:2.1 git:4.13.0 git-client:3.13.0 instance-identity:116.vf8f487400980 ionicons-api:31.v4757b_6987003 jackson2-api:2.13.4.20221013-295.v8e29ea_354141 jakarta-activation-api:2.0.1-2 jakarta-mail-api:2.0.1-2 javax-activation-api:1.2.0-5 javax-mail-api:1.6.2-5 jaxb:2.3.7-1 jdk-tool:1.0 job-dsl:1.81 jsch:0.1.55.61.va_e9ee26616e7 mailer:438.v02c7f0a_12fa_4 pipeline-build-step:2.18 pipeline-groovy-lib:613.v9c41a_160233f pipeline-input-step:456.vd8a_957db_5b_e9 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2118.v31fd5b_9944b_5 pipeline-model-definition:2.2118.v31fd5b_9944b_5 pipeline-model-extensions:2.2118.v31fd5b_9944b_5 pipeline-stage-step:296.v5f6908f017a_5 pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5 pipeline-utility-steps:2.13.1 plain-credentials:139.ved2b_9cf7587b scm-api:621.vda_a_b_055e58f7 script-security:1189.vb_a_b_7c8fd5fde snakeyaml-api:1.32-86.ve3f030a_75631 ssh-credentials:305.v8f4381501156 sshd:3.236.ved5e1b_cb_50b_2 structs:324.va_f5d6774f3a_d trilead-api:2.72.v2a_3236754f73 variant:59.vf075fe829ccb workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1200.v8005c684b_a_c6 workflow-basic-steps:994.vd57e3ca_46d24 workflow-cps:2803.v1a_f77ffcc773 workflow-durable-task-step:1210.va_1e5d77e122b workflow-job:1249.v7d974144cc14 workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ```

I used to have JFR running locally and I used it to test my Jenkins Library. Recently I have updated both the Jenkins version and the JFR version. Now it's not working.

Here is the full command inside my custom JFR container:

jenkins@ba43b64d4fbc:~$ /app/bin/jenkinsfile-runner run -w /usr/share/jenkins/ -p /usr/share/jenkins/ref/plugins --withInitHooks /usr/share/jenkins/ref/init.groovy.d/ -f /workspace/logging/logging_example.groovy 
java.lang.RuntimeException: Unhandled exception
        at io.jenkins.jenkinsfile.runner.bootstrap.commands.JenkinsLauncherCommand.call(JenkinsLauncherCommand.java:69)
        at io.jenkins.jenkinsfile.runner.bootstrap.commands.JenkinsLauncherCommand.call(JenkinsLauncherCommand.java:37)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
        at picocli.CommandLine.execute(CommandLine.java:2078)
        at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.main(Bootstrap.java:46)
Caused by: java.lang.NoClassDefFoundError: org/eclipse/jetty/util/thread/ThreadPool
        at io.jenkins.jenkinsfile.runner.App.run(App.java:25)
        at io.jenkins.jenkinsfile.runner.bootstrap.commands.JenkinsLauncherCommand.runJenkinsfileRunnerApp(JenkinsLauncherCommand.java:226)
        at io.jenkins.jenkinsfile.runner.bootstrap.commands.JenkinsLauncherCommand.call(JenkinsLauncherCommand.java:67)
        ... 9 more
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.util.thread.ThreadPool
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 12 more

Here is the DockerFile used to build my custom JFR image.

ARG baseImage=dsty/jenkins

FROM jenkins/jenkinsfile-runner:latest as jfr

FROM ${baseImage}

COPY --from=jfr /app /app

USER root

RUN cd /usr/share/jenkins && jar -xvf jenkins.war

ENV JENKINS_HOME="/usr/share/jenkins/ref/"
ENV JAVA_OPTS="-Djenkins.model.Jenkins.slaveAgentPort=50000 -Djenkins.model.Jenkins.slaveAgentPortEnforce=true -Dhudson.model.LoadStatistics.clock=1000"

USER jenkins

ENTRYPOINT ["/app/bin/jenkinsfile-runner", "run", "-w", "/usr/share/jenkins/", "-p", "/usr/share/jenkins/ref/plugins", "--withInitHooks", "/usr/share/jenkins/ref/init.groovy.d/", "-f"]

CMD ["/workspace/Jenkinsfile"]

Note the ARG baseImage=dsty/jenkins is just the FROM docker.io/jenkins/jenkins:latest-jdk11 image with some additional apt packages and plugins installed.

What Operating System are you using (both controller, and any agents involved in the problem)?

jenkins/jenkins:latest-jdk11 I think its ubuntu/debian based?

Reproduction steps

Well... my setup is a bit advanced. Probably the same with everyone who uses JFR. My code is all opensource. I could create a branch just with my docker changes and open a PR. That would make it easy to see what was changed and for me to come back here and create reproduction steps. I will do this if anyone is interested it trying to troubleshoot.

Okay.. so I can confirm that changing either the JFR version or the Jenkins version broke things because all of my CICD jobs are failing. Broke around April 5th.

Expected Results

Well usually it executes my Jenkinsfile =)

Actual Results

I got strange error about ThreadPools.

Anything else?

I <3 JFR and getting this backup and running is a huge deal for my project. https://github.com/DontShaveTheYak/jenkins-std-lib

shadycuz commented 1 year ago

oh and here is the exploded war directory:

root@b98216cc7f29:/var/jenkins_home# ls -la /usr/share/jenkins/
total 91460
drwxr-xr-x 12 root    root     4096 Nov  2 07:27 .
drwxr-xr-x 55 root    root     4096 Nov  2 07:14 ..
drwxr-xr-x  3 root    root     4096 Nov  2 07:27 META-INF
drwxr-xr-x  6 root    root     4096 Nov  1 23:51 WEB-INF
-rw-r--r--  1 root    root    20643 Nov  1 23:48 apple-touch-icon.png
drwxr-xr-x  3 root    root     4096 Nov  1 23:51 bootstrap
drwxr-xr-x  5 root    root     4096 Nov  1 23:51 css
drwxr-xr-x  2 root    root     4096 Nov  1 23:51 executable
-rw-r--r--  1 root    root    17542 Nov  1 23:48 favicon.ico
-rw-r--r--  1 root    root    36275 Nov  1 23:48 favicon.svg
drwxr-xr-x 12 root    root     4096 Nov  1 23:51 help
drwxr-xr-x  9 root    root     4096 Nov  1 23:51 images
-rw-r--r--  1 root    root 93469865 Nov  2 06:31 jenkins.war
drwxr-xr-x  4 root    root     4096 Nov  1 23:51 jsbundles
-rw-r--r--  1 root    root    45846 Nov  1 23:48 mask-icon.svg
drwxr-xr-x  4 jenkins root     4096 Nov  2 07:17 ref
-rw-r--r--  1 root    root       71 Nov  1 23:48 robots.txt
drwxr-xr-x  3 root    root     4096 Nov  1 23:51 scripts

I have tried the jenkinsfile-runner run command as both jenkins and root but same error.

I have also tried not using the -w flag but that results in this:

root@b98216cc7f29:/var/jenkins_home# /app/bin/jenkinsfile-runner run -p /usr/share/jenkins/ref/plugins --withInitHooks /usr/share/jenkins/ref/init.groovy.d/ -f /workspace/logging/logging_example.groovy 
No explicit version has been selected, using latest LTS
Running pipeline on jenkins 2.361.2
Downloading jenkins 2.361.2...
Following redirect...

It seems to be stuck there? I think its been 10 mins or so but I will let it go another 10.

Edit:

Eventually it continued and then failed with the same error:

Exploding /root/.jenkinsfile-runner/war/2.361.2/jenkins-war-2.361.2.war, this might take some time.
java.lang.RuntimeException: Unhandled exception
        at io.jenkins.jenkinsfile.runner.bootstrap.commands.JenkinsLauncherCommand.call(JenkinsLauncherCommand.java:69)
        at io.jenkins.jenkinsfile.runner.bootstrap.commands.JenkinsLauncherCommand.call(JenkinsLauncherCommand.java:37)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
        at picocli.CommandLine.execute(CommandLine.java:2078)
        at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.main(Bootstrap.java:46)
Caused by: java.lang.NoClassDefFoundError: org/eclipse/jetty/util/thread/ThreadPool
        at io.jenkins.jenkinsfile.runner.App.run(App.java:25)
        at io.jenkins.jenkinsfile.runner.bootstrap.commands.JenkinsLauncherCommand.runJenkinsfileRunnerApp(JenkinsLauncherCommand.java:226)
        at io.jenkins.jenkinsfile.runner.bootstrap.commands.JenkinsLauncherCommand.call(JenkinsLauncherCommand.java:67)
        ... 9 more
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.util.thread.ThreadPool
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 12 more
shadycuz commented 1 year ago

Java version in case someone asks?

root@b98216cc7f29:/var/jenkins_home# java --version
openjdk 11.0.16.1 2022-08-12
OpenJDK Runtime Environment Temurin-11.0.16.1+1 (build 11.0.16.1+1)
OpenJDK 64-Bit Server VM Temurin-11.0.16.1+1 (build 11.0.16.1+1, mixed mode)
shadycuz commented 1 year ago

Closing this in favor of #657