search

timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-39669] Builds fail when loading jenkins.util.SystemProperties caused by java.lang.NoClassDefFoundError: javax/servlet/ServletContextListener #3678

Closed timja closed 6 years ago

timja commented 7 years ago

We have a similar exception as in JENKINS-35184 but this happens with builds only on master. In the hope that it is fixed we upgraded to 2.19.2 but it didn't help.

The exception occurs on a random basis and cannot be pinpointed to a specific build, plugin, time,..
When it happens, all builds after that tend to fail as well. We currently have to do a restart of Jenkins on a daily basis and sometimes multiple times.

Restarting Jenkins is the only workaround we currently have and is growing into a pain especially when we are releasing. This issue is a blocker for us.

Our setup is:

StackTrace from the build logs:

java.io.IOException: Remote call on Channel to Maven [/opt/jdk1.8.0_112/bin/java, -cp, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven33-agent-1.8.1.jar:/opt/apache-maven-3.3.9/boot/plexus-classworlds-2.5.2.jar:/opt/apache-maven-3.3.9/conf/logging, jenkins.maven3.agent.Maven33Main, /opt/apache-maven-3.3.9, /var/cache/jenkins/war/WEB-INF/lib/remoting-2.62.jar, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven33-interceptor-1.8.1.jar, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-commons-1.8.1.jar, 36506] failed
    at hudson.remoting.Channel.call(Channel.java:805)
    at hudson.maven.ProcessCache$MavenProcess.call(ProcessCache.java:161)
    at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:873)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
    at hudson.model.Run.execute(Run.java:1720)
    at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:544)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:404)
Caused by: java.lang.LinkageError: Failed to load jenkins.util.SystemProperties
    at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:377)
    at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:285)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at hudson.Main.(Main.java:222)
    at hudson.EnvVars.initMaster(EnvVars.java:432)
    at hudson.EnvVars.(EnvVars.java:427)
    at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
    at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1787)
    at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:72)
    at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:253)
    at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:251)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:250)
    at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:611)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
    at hudson.remoting.UserRequest.deserialize(UserRequest.java:217)
    at hudson.remoting.UserRequest.perform(UserRequest.java:131)
    at hudson.remoting.UserRequest.perform(UserRequest.java:50)
    at hudson.remoting.Request$2.run(Request.java:332)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to Channel to Maven [/opt/jdk1.8.0_112/bin/java, -cp, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven33-agent-1.8.1.jar:/opt/apache-maven-3.3.9/boot/plexus-classworlds-2.5.2.jar:/opt/apache-maven-3.3.9/conf/logging, jenkins.maven3.agent.Maven33Main, /opt/apache-maven-3.3.9, /var/cache/jenkins/war/WEB-INF/lib/remoting-2.62.jar, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven33-interceptor-1.8.1.jar, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-commons-1.8.1.jar, 36506](Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433)
    at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
    at hudson.remoting.Channel.call(Channel.java:797)
    ... 7 more
Caused by: java.lang.NoClassDefFoundError: javax/servlet/ServletContextListener
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
    at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:373)
    at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:285)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at hudson.Main.(Main.java:222)
    at hudson.EnvVars.initMaster(EnvVars.java:432)
    at hudson.EnvVars.(EnvVars.java:427)
    at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
    at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1787)
    at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:72)
    at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:253)
    at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:251)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:250)
    at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:611)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
    at hudson.remoting.UserRequest.deserialize(UserRequest.java:217)
    at hudson.remoting.UserRequest.perform(UserRequest.java:131)
    at hudson.remoting.UserRequest.perform(UserRequest.java:50)
    at hudson.remoting.Request$2.run(Request.java:332)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: javax.servlet.ServletContextListener (java.net.URLClassLoader@3f99bd52 did not find class file)
    at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch2(RemoteClassLoader.java:792)
    at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch4(RemoteClassLoader.java:842)
    at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch3(RemoteClassLoader.java:849)
    at sun.reflect.GeneratedMethodAccessor208.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:887)
    at hudson.remoting.Request$2.run(Request.java:332)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
    at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to channel(Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433)
    at hudson.remoting.Request.call(Request.java:172)
    at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:255)
    at com.sun.proxy.$Proxy2.fetch3(Unknown Source)
    at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:187)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 36 more

Installed plugins

ace-editor 1.1 true    false
active-directory    1.49    true    false
analysis-core   1.79    true    false
android-emulator    2.15    true    false
android-lint    2.4 true    false
ant 1.4 true    false
antisamy-markup-formatter   1.5 true    false
authorize-project   1.2.2   true    false
bouncycastle-api    2.16.0  true    false
branch-api  1.11    true    false
build-failure-analyzer  1.17.2  true    false
build-monitor-plugin    1.10+build.201610041454 true    false
build-name-setter   1.6.5   true    false
build-pipeline-plugin   1.5.4   true    false
build-timeout   1.17.1  true    false
build-timestamp 1.0.1   true    false
claim   2.8 true    false
cloudbees-folder    5.13    true    false
conditional-buildstep   1.3.5   true    false
config-file-provider    2.13    true    false
credentials 2.1.8   true    false
credentials-binding 1.9 true    false
dashboard-view  2.9.10  true    false
display-url-api 0.5 true    false
durable-task    1.12    true    false
external-monitor-job    1.6 true    false
git 3.0.0   true    false
git-client  2.0.0   true    false
git-parameter   0.7.0   true    false
git-server  1.7 true    false
gradle  1.25    true    false
greenballs  1.15    true    false
handlebars  1.1.1   true    false
htmlpublisher   1.11    true    false
icon-shim   2.0.3   true    false
javadoc 1.4 true    false
jobConfigHistory    2.15    true    false
jobgenerator    1.22    true    false
jquery  1.11.2-0    true    false
jquery-detached 1.2.1   true    false
junit   1.19    true    false
ldap    1.13    true    false
locale  1.2 true    false
log-parser  2.0 true    false
m2release   0.14.0  true    false
mailer  1.18    true    false
matrix-auth 1.4 true    false
matrix-project  1.7.1   true    false
maven-plugin    2.14    true    false
momentjs    1.1.1   true    false
nodejs  0.2.1   true    false
pam-auth    1.3 true    false
parameterized-trigger   2.32    true    false
pipeline-build-step 2.3 true    false
pipeline-graph-analysis 1.2 true    false
pipeline-input-step 2.3 true    false
pipeline-milestone-step 1.1 true    false
pipeline-rest-api   2.1 true    false
pipeline-stage-step 2.2 true    false
pipeline-stage-view 2.1 true    false
plain-credentials   1.3 true    false
port-allocator  1.8 true    false
protex-jenkins  1.4.0   true    false
publish-over-ssh    1.14    true    false
run-condition   1.0 true    false
scm-api 1.3 true    false
script-security 1.24    true    false
scriptler   2.9 true    false
sectioned-view  1.20    true    false
sonar   2.5 true    false
ssh-agent   1.13    true    false
ssh-credentials 1.12    true    false
ssh-slaves  1.11    true    false
stash-pullrequest-builder   1.7.0   true    false
stashNotifier   1.11.4  true    false
structs 1.5 true    false
thinBackup  1.7.4   true    false
token-macro 2.0 true    false
windows-slaves  1.2 true    false
workflow-aggregator 2.4 true    false
workflow-api    2.5 true    false
workflow-basic-steps    2.2 true    false
workflow-cps    2.21    true    false
workflow-cps-global-lib 2.4 true    false
workflow-durable-task-step  2.5 true    false
workflow-job    2.8 true    false
workflow-multibranch    2.9 true    false
workflow-scm-step   2.2 true    false
workflow-step-api   2.4 true    false
workflow-support    2.10    true    false
ws-cleanup  0.30    true    false

We added additional loggers: hudson.remoting.RemoteClassLoader and org.jenkinsci.plugins but they did not provide helpful information.

As discussed in JENKINS-35184 this might be caused by a plugin especially the maven plugin thus I set it as component as well.

If there is more information needed please state so, we will gladly help to get this fixed

Originally reported by robert, imported from: Builds fail when loading jenkins.util.SystemProperties caused by java.lang.NoClassDefFoundError: javax/servlet/ServletContextListener
  • status: Resolved
  • priority: Blocker
  • resolution: Duplicate
  • resolved: 2018-05-21T21:14:45+00:00
  • imported: 2022/01/10
timja commented 7 years ago

robert:

We found a workaround - which is not nice.

The workaround is to put the servlet-api jar (containing ServletContextListener.class) on the classpath of a Maven build. We did this by copying the jar (http://search.maven.org/remotecontent?filepath=javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar) into the /lib/ext folder of the JRE.
This is not nice but I wanted to document it here in case somebody else runs into this problem.

Some explanations:

It looks like the exception stems from a remote process and we suspect it comes from Maven builds (Jobs with Maven Project Type). As these run in their own Java Process it looks like the Systemproperties are somewhat loaded in that process but don't have the necessary dependencies.
After looking in the pom.xml on Github, we saw the dependency to the javax.servlet-api in scope "provided".
The Jar itself is contained in the winstone container but not in standard Java libraries.
As cannot figure out when this happens it was worth a try to just provide the class to see if it works or produces a different error.

Builds are stable since two days - but still .. this bugs needs to be fixed.

timja commented 7 years ago

oleg_nenashev:

it's just another issue with Maven interceptors. SystemProperties class in Jenkins requires ServletContextListener, and seems the the missing class causes fatal failure, because the class is not being sent over the channel due to security reasons IIRC.

timja commented 6 years ago

scm_issue_link:

Code changed in jenkins
User: Nicolas De Loof
Path:
docker-plugin/pom.xml
docker-plugin/src/main/java/com/nirima/jenkins/plugins/docker/utils/JenkinsUtils.java
docker-plugin/src/main/java/io/jenkins/docker/pipeline/DockerNodeStep.java
docker-plugin/src/main/java/io/jenkins/docker/pipeline/DockerNodeStepExecution.java
docker-plugin/src/main/resources/io/jenkins/docker/pipeline/DockerNodeStep/config.jelly
http://jenkins-ci.org/commit/docker-plugin/9bf481aac670959542dcb8f576e1e803b422b895
Log:
upgrade to latest Jenkins and dependencies
workaround for resurrecting JENKINS-39669

timja commented 6 years ago

jglick:

I think this case was fixed in JENKINS-46386.

timja commented 2 years ago

[Duplicates: JENKINS-46386]