search

timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-43405] NPE when loading Manage Jenkins #3988

Open timja opened 7 years ago

timja commented 7 years ago

When navigating to the Manage Jenkins page, I get this:

${h.initPageVariables(context)} Skip to content

 
 
 
  
 
Skip to content

 and a NPE in the log:
 
Apr 06, 2017 2:46:56 PM org.eclipse.jetty.util.log.JavaUtilLog warn
WARNING: /manage
javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-2.46.1.jar!/lib/layout/layout.jelly:268:155: java.lang.NullPointerException
at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:105)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at hudson.init.impl.InstallUncaughtExceptionHandler$1.reportException(InstallUncaughtExceptionHandler.java:30)
at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:77)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:55)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.commons.jelly.JellyTagException: jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-2.46.1.jar!/lib/layout/layout.jelly:268:155: java.lang.NullPointerException
at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:726)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:281)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:97)
... 29 more
Caused by: java.lang.NullPointerException
at java.util.Calendar.setTime(Calendar.java:1770)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:943)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:936)
at java.text.DateFormat.format(DateFormat.java:345)
at org.apache.commons.jelly.tags.fmt.FormatDateTag.doTag(FormatDateTag.java:182)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
... 50 more
 

Originally reported by chandau, imported from: NPE when loading Manage Jenkins
  • status: Open
  • priority: Minor
  • resolution: Unresolved
  • imported: 2022/01/10
timja commented 7 years ago

oleg_nenashev:

Code: https://github.com/jenkinsci/jenkins/blob/jenkins-2.46.1/core/src/main/resources/lib/layout/layout.jelly#L268

I have no idea what could cause that. Does it happen consistently after the master restart?

timja commented 7 years ago

chandau:

Yes unfortunately. It will work for a bit, then crash. I thought maybe it was a memory issue (even though there was no indication in the logs) so I added -Xmx2048m to the startup args. This seems to have increased the time between NPEs, but didn't fix the issue.

timja commented 7 years ago

oleg_nenashev:

Some analysis:

No idea so far. Is there a chance you have a plugin, which depends on commons-jelly and pulls in another version of the lib?

timja commented 7 years ago

chandau:

Here's the list of plugins installed:

Active Directory 2.4
AnsiColor 0.5.0
Ant 1.4
Authentication Tokens API 1.3
bouncycastle API 2.16.1
Branch API 2.0.8
Build Pipeline 1.5.6
Build Time Blame 1.1.0
build timeout 1.18
build-metrics 1.3
Cmake 2.4.5
Cobertura 1.9.8
Conditional BuildStep 1.3.5
Confluence Publisher 1.8
Copy Artifact 1.38.1
Coverage/Complexity Scatter Plot 1.1.1
Cppcheck 1.21
Credentials Binding 1.11
Credentials 2.1.13
Dashboard 2.9.10
disk-usage 0.28
Display URL 1.1.1
Docker Commons 1.6
Docker Pipelin 1.10
DocLinks 0.6.1
Durable Task 1.13
Email Extension 2.57.1
embeddable build-status 1.9
Environment Injector 1.93.1
External Monitor Job Type 1.7
Folders 6.0.3
Git Changelog 1.48
Git client 2.4.1
Git 3.2.0
GIT server 1.7
GitHub API 1.85
GitHub Branch Source 2.0.5
GitHub Integration 0.1.0-rc22
GitHub Organization Folder 1.6
GitHub 1.26.2
GitHub Pull Request Builder 1.35.0
GitHub Pull Request Coverage Status 1.6.0
Gradle 1.26
Green Balls 1.15
HTML Publisher 1.12
Hudson global-build-stats 1.4
Icon Shim 2.0.3
Jackson 2 API 2.7.3
Javadoc 1.4
JavaScript GUI Lib: ACE Editor bundle 1.1
JavaScript GUI Lib: Handlebars bundle 1.1.1
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) 1.2.1
JavaScript GUI Lib: Moment.js bundle 1.1.1
Jira Issue Updated 1.18
JIRA 2.3
jira-ext 0.6.1
Job Configuration History 2.15
Job Import 1.6
jQuery 1.11.2-0
JUnit 1.20
Keychains and Provisioning Profiles Management 1.0.0
LDAP 1.14
Log Parser 2.0
Mailer 1.20
MapDB API 1.0.9.0
Matrix Authorization Strategy 1.4
Matrix Project 1.9
Matrix Reloaded 1.1.3
Maven Integration 2.15.1
Metrics Disk Usage 3.0.0
Metrics 3.1.2.9
Monitoring 1.65.1
Multiple SCMs 0.6
OWASP Markup Formatter 1.5
PAM Authentication 1.3
Parameterized Trigger 2.33
Pipeline 2.5
Pipeline GitHub Notify Step 1.0.1
Pipeline Graph Analysis 1.3
Pipeline: API 2.12
Pipeline: Basic Steps 2.4
Pipeline: Build Step 2.4
Pipeline: Declarative Extension Points API 1.1.1
Pipeline: GitHub Groovy Libraries 1.0
Pipeline: Groovy 2.29
Pipeline: Inpu Step 2.5
Pipeline: Job 2.10
Pipeline: Milestone Step 1.3.1
Pipeline: Model API 1.1.1
Pipeline: Model Definition 1.1.1
Pipeline: Multibranch 2.14
Pipeline: Nodes and Processes 2.10
Pipeline: REST API 2.6
Pipeline: SCM Step 2.4
Pipeline: Shared Groovy Libraries 2.7
Pipeline: Stage Step 2.2
Pipeline: Stage Tags Metadata 1.1.1
Pipeline: Stage View 2.6
Pipeline: Step API 2.9
Pipeline: Supporting APIs 2.14
Plain Credentials 1.4
Plugin Usage 0.3
Project Build Times 1.2.1
Publish Over CIFS 0.3
Rebuilder 1.25
Release Helper 1.3
Resource Disposer 0.6
Run Condition 1.0
SCM API 2.1.1
Script Security 1.27
SLOCCount 1.21
SSH Agent 1.15
SSH Credentials 1.13
SSH Slaves 1.16
Structs 1.6
Subcersion 2.7.2
ThinBackup 1.9
Timestamper 1.8.8
Token Macro 2.1
Valgrind 0.27
Windows Slaves 1.3.1
Workspace Cleanup 0.32
Xcode integration 1.4.11
xUnit 1.102

timja commented 7 years ago

chandau:

Still happening, seems to not be any specific page load, and some of the plugins have been updated since I last updated this bug. Is there something else we can try? this is almost making Jenkins unusable at this point

Apr 21, 2017 1:22:20 PM org.eclipse.jetty.util.log.JavaUtilLog warn
WARNING: /view/MARS/job/unifiedmesh-multi-release/configure
javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-2.46.1.jar!/lib/layout/layout.jelly:268:155: java.lang.NullPointerException
at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:105)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at hudson.init.impl.InstallUncaughtExceptionHandler$1.reportException(InstallUncaughtExceptionHandler.java:30)
at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:77)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:55)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)

...

Caused by: org.apache.commons.jelly.JellyTagException: jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-2.46.1.jar!/lib/layout/layout.jelly:268:155: java.lang.NullPointerException

 

timja commented 7 years ago

chandau:

Still happening daily, but changing the startup options to  -Xmx2048m seems to have reduced the number of times we have to restart Jenkins in a day (now it's only 2-3 times, vs every hour or so)

timja commented 7 years ago

oleg_nenashev:

Would it be possible to get System and GC logs from the instance? Maybe there is a Weak reference somewhere in the logic.

timja commented 7 years ago

chandau:

I can do system logs, but will need to enable gc logs as I don't see that they were enabled before (unless I'm missing something)

timja commented 7 years ago

chandau:

ok this will sound weird, but ever since I put in the java options for gc, it's been more responsive and I haven't been able to get an npe (I'm hoping either it was that easy or I can jinx myself here and have it crash once I post this )

 

timja commented 7 years ago

chandau:

Well - I don't see anything useful in the jenkins.log, but I do get this when trying to configure a job

timja commented 7 years ago

chandau:

Now it's worse in that I don't get a stack trace in the log, just on screen when navigating to Manage Jenkins. I had upgraded to 2.46.2 in hopes that something would improve.

The gc log is just full of:

2017-05-04T09:52:24.147-0700: 116.969: [GC (Allocation Failure) 725206K->87054K(821760K), 0.0081444 secs]
2017-05-04T09:52:36.370-0700: 129.191: [GC (Allocation Failure) 731662K->104461K(822272K), 0.0258021 secs]

with no indication of any difference in the entry when Jenkins is working vs what it's doing right now, which is not letting me navigate to any configuration for any job or the manage jenkins page.

timja commented 7 years ago

oleg_nenashev:

So, I would start from switching to another GC/JVM

timja commented 7 years ago

chandau:

I had been upgrading whenever there was a release, but that didn't change the behavior. I've switched from openjdk 1.8.0_131-b12 to oracle 1.8.0_131-b11 & will post the results

timja commented 7 years ago

chandau:

well it is slightly more stable with the oracle jdk, but it still crashes at least daily

current versions:

jenkins 2.60.1

oracle jdk 1.8.0_131

It's not a very large installation, only 31 jobs, 4 active nodes, and no jobs on master

timja commented 7 years ago

chandau:

Any other suggestions? Still getting npes with v2.60.2

Now it sometimes doesn't fully come back after restarting or rebooting the server

INFO: Failed to instantiate optional component hudson.plugins.build_timeout.operations.AbortAndRestartOperation$DescriptorImpl; skipping
Aug 01, 2017 9:22:10 AM net.bull.javamelody.internal.common.JavaLogger warn
WARNING: exception while collecting data: java.lang.NullPointerException
java.lang.NullPointerException
at net.bull.javamelody.internal.model.Collector.addJRobinValue(Collector.java:627)
at net.bull.javamelody.internal.model.Collector.collectCounterData(Collector.java:710)
at net.bull.javamelody.internal.model.Collector.collect(Collector.java:351)
at net.bull.javamelody.internal.model.Collector.collectWithoutErrors(Collector.java:319)
at net.bull.javamelody.NodesCollector.collectWithoutErrorsNow(NodesCollector.java:173)
at net.bull.javamelody.NodesCollector.collectWithoutErrors(NodesCollector.java:147)
at net.bull.javamelody.NodesCollector$1.run(NodesCollector.java:91)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

timja commented 7 years ago

chandau:

upgraded java to 1.8.0_144 and now get 

${h.printThrowable(request.getAttribute('javax.servlet.error.exception'))}

which isn't terrible helpful and still restarting jenkins several times a day

timja commented 7 years ago

chandau:

and now I get this after a restart - should I just give up and go with circle-ci or travis?

changing to an attachment as it won't let me paste screen shot

timja commented 7 years ago

oleg_nenashev:

NPE in JavaMelody is definitely an issue in Monitoring plugin. Makes sense to create a ticket for it though it unlikely has anything to do with the showstopper issue reported here.

Regarding "st:include should never happen", I need a full stacktrace for the issue. It seems to be an assert in a plugin somewhere

timja commented 7 years ago

chandau:

This is all I have in the log, will look at creating separate issue for monitoring, but if that's not what's creating the instability, it's a much lower priority:

Aug 08, 2017 4:58:01 PM hudson.init.impl.InstallUncaughtExceptionHandler$1 reportException

WARNING: null

javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-2.60.2.jar!/lib/form/property.jelly:40:72: No page found '${descriptor.configPage}' for class java.lang.String

        at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:105)

        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)

        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)

        at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)

        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)

        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)

        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)

        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)

        at org.kohsuke.stapler.Stapler.service(Stapler.java:238)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)

        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)

        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:237)

        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:209)

        at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)

        at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:113)

        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)

        at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)

        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)

        at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)

        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)

        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:86)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)

        at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

        at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

        at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

        at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

        at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

        at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

        at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)

        at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)

        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

        at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

        at org.eclipse.jetty.server.Server.handle(Server.java:499)

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

        at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.commons.jelly.JellyTagException: jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-2.60.2.jar!/lib/form/property.jelly:40:72: No page found '${descriptor.configPage}' for class java.lang.String

        at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:124)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

        at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

        at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)

        at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

        at org.apache.commons.jelly.tags.core.WhenTag.doTag(WhenTag.java:46)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

        at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

        at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

        at org.apache.commons.jelly.tags.core.WhenTag.doTag(WhenTag.java:46)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

        at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)

        at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)

        at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)

        at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)

        at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:97)

        ... 68 more

 

timja commented 7 years ago

oleg_nenashev:

"No page found '${descriptor.configPage}' for class java.lang.String" means that there is a plugin misusing the f:property control provided by Jenkins. danielbeck does Stapler trace mode help with such form processing. And it contributes to the "Configure" page...

Please provide a list of the installed plugins. Do you also have any custom plugins installed?

timja commented 7 years ago

chandau:

No custom plugins - all installed via https://updates.jenkins-ci.org/stable/update-center.json

ace-editor
active-directory
ansicolor
ant
antisamy-markup-formatter
artifact-promotion
authentication-tokens
bouncycastle-api
branch-api
build-metrics
build-pipeline-plugin
build-time-blame
build-timeout
cloudbees-folder
cmakebuilder
cobertura
conditional-buildstep
confluence-publisher
copyartifact
covcomplplot
cppcheck
credentials
credentials-binding
dashboard-view
depgraph-view
disk-usage
display-url-api
docker-commons
docker-workflow
doclinks
durable-task
email-ext
emailext-template
embeddable-build-status
envinject
envinject-api
external-monitor-job
ghprb
git
git-changelog
git-client
github
github-api
github-branch-source
github-organization-folder
github-pr-coverage-status
github-pullrequest
git-server
global-build-stats
gradle - disabled
greenballs
handlebars
htmlpublisher
icon-shim
jackson2-api
javadoc
jenkins-jira-issue-updater - disabled
jira
jira-ext
jobConfigHistory
job-import-plugin
jquery
jquery-detached

jquery-ui
junit
kpp-management-plugin
ldap
log-parser
mailer
mapdb-api
matrix-auth
matrix-project
matrix-reloaded
maven-plugin
metrics
metrics-diskusage
momentjs
monitoring
multiple-scms - disabled
pam-auth
parameterized-trigger
pipeline-build-step
pipeline-github-lib

pipeline-githubnotify-step

pipeline-graph-analysis
pipeline-input-step
pipeline-milestone-step
pipeline-model-api
pipeline-model-declarative-agent
pipeline-model-definition
pipeline-model-extensions
pipeline-rest-api
pipeline-stage-step
pipeline-stage-tags-metadata
pipeline-stage-view
plain-credentials
plugin-usage-plugin
project-build-times
promoted-builds
publish-over-cifs
rebuild
release-helper
resource-disposer
run-condition
scm-api

script-security
sloccount
ssh-agent
ssh-credentials
ssh-slaves
structs
subversion
thinBackup
timestamper
token-macr
valgrind
windows-slaves
workflow-aggregator
workflow-api
workflow-basic-steps
workflow-cps
workflow-cps-global-lib
workflow-durable-task-step
workflow-job
workflow-multibranch
workflow-scm-step

workflow-step-api
workflow-support
ws-cleanup
xcode-plugin
xunit

 

timja commented 7 years ago

danielbeck:

Something's really messed up here, the formatDate calls h.getCurrentTime() which is just return new Date().

My best guess based on other output that something's messed up with JEXL expressions. Pure guess, and no idea what or why.

timja commented 7 years ago

danielbeck:

Was Jenkins restarted since this was first occurring?

timja commented 7 years ago

chandau:

Yes - many times, as it becomes completely useless once it happens

timja commented 7 years ago

danielbeck:

chandau To clarify, this does not occur immediately after startup? For how long does Jenkins work between starting and this error occurring?

timja commented 7 years ago

chandau:

It varies - seems to depend on how many folks are trying to access the website. Sometimes it fails to start and takes a few restarts or a system reboot to come back on line. Believe me, I would much rather it be easily reproducible.

timja commented 7 years ago

chandau:

ok so just restarted Jenkins because of this error, and it happened again less than 2 minutes of running, by trying to configure an existing job. I have restarted 3 times now trying to configure this job.

timja commented 7 years ago

chandau:

Not sure if this is a related issue, but after getting garbage on the screen, I tried restarting jenkins and am getting this in the log:

Aug 22, 2017 9:07:18 AM net.bull.javamelody.internal.common.JavaLogger warn
WARNING: exception while collecting data: java.lang.NullPointerException
java.lang.NullPointerException
at net.bull.javamelody.internal.model.Collector.addJRobinValue(Collector.java:627)
at net.bull.javamelody.internal.model.Collector.collectCounterData(Collector.java:710)
at net.bull.javamelody.internal.model.Collector.collect(Collector.java:351)
at net.bull.javamelody.internal.model.Collector.collectWithoutErrors(Collector.java:319)
at net.bull.javamelody.NodesCollector.collectWithoutErrorsNow(NodesCollector.java:173)
at net.bull.javamelody.NodesCollector.collectWithoutErrors(NodesCollector.java:147)
at net.bull.javamelody.NodesCollector$1.run(NodesCollector.java:91)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

timja commented 7 years ago

chandau:

And this after restarting the computer:

java.lang.RuntimeException: Should never happen at org.codehaus.groovy.reflection.CachedMethod.compareToCachedMethod(CachedMethod.java:174) at org.codehaus.groovy.reflection.CachedMethod.compareTo(CachedMethod.java:140) at java.util.ComparableTimSort.countRunAndMakeAscending(ComparableTimSort.java:320) at java.util.ComparableTimSort.sort(ComparableTimSort.java:202) at java.util.Arrays.sort(Arrays.java:1246) at org.codehaus.groovy.reflection.CachedClass$3.initValue(CachedClass.java:136) at org.codehaus.groovy.reflection.CachedClass$3.initValue(CachedClass.java:102) at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:49) at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:36) at org.codehaus.groovy.reflection.CachedClass.getMethods(CachedClass.java:274) at groovy.lang.MetaClassImpl.populateMethods(MetaClassImpl.java:396) at groovy.lang.MetaClassImpl.fillMethodIndex(MetaClassImpl.java:346) at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:3286) at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:271) at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:313) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:258) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:883) at groovy.lang.GroovyObjectSupport.(GroovyObjectSupport.java:34) at groovy.lang.GroovyShell.(GroovyShell.java:89) at hudson.util.spring.BeanBuilder.parse(BeanBuilder.java:134) at hudson.plugins.active_directory.ActiveDirectorySecurityRealm.createSecurityComponents(ActiveDirectorySecurityRealm.java:332) at hudson.security.SecurityRealm.getSecurityComponents(SecurityRealm.java:454) at hudson.security.HudsonFilter.reset(HudsonFilter.java:138) at jenkins.model.Jenkins.setSecurityRealm(Jenkins.java:2578) at jenkins.model.Jenkins$19.run(Jenkins.java:3179) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:1090) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused: org.jvnet.hudson.reactor.ReactorException at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:269) at jenkins.InitReactorRunner.run(InitReactorRunner.java:47) at jenkins.model.Jenkins.executeReactor(Jenkins.java:1124) at jenkins.model.Jenkins.(Jenkins.java:929) at hudson.model.Hudson.(Hudson.java:86) at hudson.model.Hudson.(Hudson.java:82) at hudson.WebAppMain$3.run(WebAppMain.java:235) Caused: hudson.util.HudsonFailedToLoad at hudson.WebAppMain$3.run(WebAppMain.java:252)