[JENKINS-29078] hudson.triggers.Trigger$Cron doRun throwing a NullPointerException

[timja]

Hi, it seems that Jenkins is throwing a NullPointerException on the below in the background logs but i can't seem to tell why.

Jun 25, 2015 5:38:00 PM hudson.triggers.Trigger$Cron doRun
WARNING: Cron thread throw an exception
java.lang.NullPointerException
at hudson.model.Descriptor.toMap(Descriptor.java:880)
at hudson.util.DescribableList.toMap(DescribableList.java:151)
at hudson.model.AbstractProject.getTriggers(AbstractProject.java:1615)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:215)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

---

Originally reported by chinboon, imported from: hudson.triggers.Trigger$Cron doRun throwing a NullPointerException

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[timja]

phild:

None of our cron scheduled tasks are running!
This seems pretty serious.

[timja]

phild:

Downgrading to 1.615 fixes things for us.

[timja]

phild:

Looks like JENKINS-27549 is the probable cause

[timja]

chinboon:

Hi danielbeck, i tried the command but it is not displaying anything.

[timja]

danielbeck:

Sorry about that. Please try:

Jenkins.instance.getAllItems(AbstractProject).each { p ->
  try {
    p.getTriggers()
  } catch (NullPointerException npe) {
    println p.fullName
  }
}
return

[timja]

lzr:

Actually, downgrading to 1.615 didn't help. I still observe the problem. Ran the script and it printed a name of a Jenkins job. I tried saving it today and got an error page (with evil Jenkins).

What do I do now?

config.xml:

'1.0' encoding='UTF-8'?>

    
    
    false
    
    "hudson.scm.NullSCM"/>
    true
    false
    false
    false
    

    
    false
    

    sudo -u nobody script.sh path
    

    
    
"mailer@1.15">
    ops-alert@example.com
    false
    false

    
    

When I tried to re-save that project I got this exception:

javax.servlet.ServletException: java.lang.NullPointerException
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:123)
    at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120)
    at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:117)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:114)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
    at org.eclipse.jetty.ajp.Ajp13Connection.access$2900(Ajp13Connection.java:45)
    at org.eclipse.jetty.ajp.Ajp13Connection$RequestHandler.headerComplete(Ajp13Connection.java:237)
    at org.eclipse.jetty.ajp.Ajp13Parser.parseNext(Ajp13Parser.java:505)
    at org.eclipse.jetty.ajp.Ajp13Parser.parseAvailable(Ajp13Parser.java:158)
    at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
    at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
    at hudson.model.AbstractProject.submit(AbstractProject.java:1873)
    at hudson.model.Project.submit(Project.java:230)
    at hudson.model.Job.doConfigSubmit(Job.java:1206)
    at hudson.model.AbstractProject.doConfigSubmit(AbstractProject.java:788)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
    at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    ... 67 more

[timja]

danielbeck:

Hm. Weird. Difficult to say how a null trigger got in there. Do you have the Job Config History Plugin that could show you information on how the job config was changed to add that element?

You should be able to fix this by changing the XML to instead of what that element is currently while Jenkins is stopped.

[timja]

lzr:

I do have Job Config History installed but I already deleted the project I'm not sure if config history is kept after deletion.

Let me ask this though. Why does a single project with incorrect config.xml prevent other projects from being scheduled. This seems very wrong to me.

[timja]

tonyfinn:

Observed this occur on 1.620 with a Build Flow job. We didn't restart but it did prevent SCM polling jobs from running. Don't have any more useful log messages because we didn't have the config history plugin and the messages were the same as in this thread.

[timja]

integer:

tonyfinn what you describe sounds like other issue that already fixed.

chinboon without knowing what/when/who inserted null in triggers it impossible to reproduce.

[timja]

lzr:

The issue happened again, this time the config history plugin is installed and it shows that the job haven't been modified since 2015. But the cron stopped running a couple of days ago (even though it should be running every minute).

[timja]

lzr:

Nevermind. Was looking at the wrong job. Ran the script provided by danielbeck and it printed out the name of the right job that had a null trigger in config.xml.

The configuration history plugin shows two changes yesterday, with 1 minute difference (see attachments).

Untitled spreadsheet - Sheet1 (1).pdf Untitled spreadsheet - Sheet2.pdf

The user says he was editing other jobs at that time but doesn't recall modifying this particular one. But even if he edited it by mistake, it's still not clear what caused Jenkins to save null triggers, as well as remove a bunch of other configuration options.

Even if we can't reproduce this issue in full details, can you guys add a check for a null trigger? It's REALLY bad that one job with messed up configuration file causes the whole system to intermittently skip some scheduled jobs. The issue should be isolated to one job.

[timja]

[Originally duplicated by: JENKINS-31624]

[timja]

[Originally related to: JENKINS-36232]