[JENKINS-42096] SEVERE: Failed to rotate log

[timja]

In global logs I find this SEVERE :

When I check the Job, I try to manually remove the build, I have also an error message (see bellow).
All builds are successfull and I checked all locked files on servers there none of them.
I also checked the thread list in /monitoring and found nothing related...

Please advice.

INFO: HMP/HMP_DEV_TRUNK_CpoeMe #124 main build action completed: SUCCESS
Feb 09, 2017 6:44:18 PM hudson.model.Run execute
SEVERE: Failed to rotate log
java.io.IOException: K:\JENKINS\jobs\HMP\jobs\HMP_DEV_TRUNK_CpoeMe\builds\104 is in use
at hudson.model.Run.delete(Run.java:1511)
at hudson.maven.MavenModuleSetBuild.delete(MavenModuleSetBuild.java:451)
at hudson.tasks.LogRotator.perform(LogRotator.java:131)
at hudson.model.Job.logRotate(Job.java:474)
at hudson.maven.MavenModuleSet.logRotate(MavenModuleSet.java:896)
at hudson.model.Run.execute(Run.java:1792)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:544)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:404)

Manual removal :

Deletion of the build failed
Retry delete
Show reason
java.io.IOException: K:\JENKINS\jobs\HMP\jobs\HMP_DEV_TRUNK_CpoeMe\builds\104 is in use
at hudson.model.Run.delete(Run.java:1511)
at hudson.maven.MavenModuleSetBuild.delete(MavenModuleSetBuild.java:451)
at hudson.model.Run.doDoDelete(Run.java:2193)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:335)
at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:52)
at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:175)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:108)
at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:362)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:236)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:236)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:219)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:196)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)
at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:110)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:95)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

---

Originally reported by fredericmeyrou, imported from: SEVERE: Failed to rotate log

• status: Resolved
• priority: Minor
• resolution: Not A Defect
• resolved: 2017-02-20T14:00:27+00:00
• imported: 2022/01/10

[timja]

danielbeck:

This is an issue tracker, not a support site. And for a bug report, this does not contain enough information for further investigation.

[timja]

robsimon:

danielbeck

what would be needed for a proper issue? We're seeing the same issue on Jenkins v2.51 (Java 1.8.0_25-b17 64bit on Ubuntu 14.04.2 LTS) and there are strong indications that this causes our Jenkins to spike CPU usage until it becomes unresponsive...

We found a work-around... All job runs which have this issue we make a dummy directory within jobs builds home directory. After that Jenkins can clean it up and CPU usage goes down.

[timja]

danielbeck:

All job runs which have this issue we make a dummy directory within jobs builds home directory. After that Jenkins can clean it up and CPU usage goes down.

Please be more specific what you're doing.

Are you using a custom builds directory?

[timja]

robsimon:

we ended up writing a little script which we let run in the directory of the job in question as the amount of directories was in thousands...

 

#!/bin/bash

for i in {1..50000};do

    res=$(curl --write-out "%{http_code}\n" --silent --output /dev/null -u USER:PW -IL http://MASTER:8080/job/JOBNAME/$i/)

    if [[ "${res}" == "200" ]]; then

        if [ ! -d "${i}" ]; then

            echo "Make dummy for ${i}"

            mkdir ${i}

        fi

    fi

done

 

I haven't set up our Jenkins so I don't know if it is the default for Ubuntu. However, it keeps the job/builds under /local/jenkins/builds/JOBNAME/ And in this we run the script from above to create directories for which Jenkins believes there should be directories.