search

timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-42396] A lot of errors at jenkins start since upgrades to new branch models : Failed to update org.jenkinsci.plugins.workflow.job.WorkflowJob / Unable to delete ... lastSuccessfulBuild #3897

Closed timja closed 7 years ago

timja commented 7 years ago

At each jenkins restart, there are a lot of operations in error. always the same error :

java.nio.file.DirectoryNotEmptyException on 'lastSuccessfulBuild' folder deletion attempt. It seems that the deletion tries to delete a file (hudson.Util.deleteFile) but it's a folder.

This is causing heavy operations on disk and slows down a lot jenkins start. I think that this happens since the automatic automatic migration of all jobs to the SCM API 2.0 model. it seems that migration has stucked those folders.

Here is a log of one of those :



mars 01, 2017 12:08:25 PM jenkins.model.PeepholePermalink updateCache

WARNING: Failed to update org.jenkinsci.plugins.workflow.job.WorkflowJob@167c3323[DataViz/DashboardWebApp/Dashboard-Deploy/story_MS_box_multi_selection] lastSuccessfulBuild permalink for DataViz/DashboardWebApp/Dashboard-Deploy/story_MS_box_multi_selection #19

java.io.IOException: Unable to delete 'E:\Jenkins\jobs\DataViz\jobs\DashboardWebApp\jobs\Dashboard-Deploy\branches\story-MS-box.gumkf1.ti-selection\builds\lastSuccessfulBuild'. Tried 3 times (of a maximum of 3) waiting 0,1 s between attempts.

    at hudson.Util.deleteFile(Util.java:248)

    at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:113)

    at jenkins.model.PeepholePermalink.writeSymlink(PeepholePermalink.java:200)

    at jenkins.model.PeepholePermalink.updateCache(PeepholePermalink.java:150)

    at jenkins.model.PeepholePermalink.resolve(PeepholePermalink.java:122)

    at hudson.model.Job.getLastSuccessfulBuild(Job.java:912)

    at htmlpublisher.workflow.WorkflowActionsFactory.createFor(WorkflowActionsFactory.java:57)

    at htmlpublisher.workflow.WorkflowActionsFactory.createFor(WorkflowActionsFactory.java:43)

    at hudson.model.Actionable.createFor(Actionable.java:111)

    at hudson.model.Actionable.getAction(Actionable.java:334)

    at jenkins.branch.ItemColumn.isPrimary(ItemColumn.java:71)

    at sun.reflect.GeneratedMethodAccessor866.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.lang.reflect.Method.invoke(Unknown Source)

    at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)

    at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)

    at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)

    at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)

    at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)

    at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)

    at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)

    at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)

    at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsBoolean(ExpressionSupport.java:71)

    at org.apache.commons.jelly.tags.core.WhenTag.doTag(WhenTag.java:44)

    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

    at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)

    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)

    at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)

    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

    at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

    at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)

    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

    at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)

    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

    at org.kohsuke.stapler.jelly.groovy.JellyBuilder.doInvokeMethod(JellyBuilder.java:276)

    at org.kohsuke.stapler.jelly.groovy.Namespace$ProxyImpl.invoke(Namespace.java:92)

    at com.sun.proxy.$Proxy107.projectView(Unknown Source)

    at lib.JenkinsTagLib$projectView$5.call(Unknown Source)

    at hudson.model.View.main.run(main.groovy:16)

    at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:74)

    at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:62)

    at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)

    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

    at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

    at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

    at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)

    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)

    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)

    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)

    at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)

    at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:140)

    at org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:30)

    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)

    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)

    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:782)

    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)

    at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)

    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)

    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)

    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)

    at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)

    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)

    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)

    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)

    at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)

    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)

    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)

    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)

    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)

    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)

    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)

    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)

    at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)

    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)

    at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)

    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)

    at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)

    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)

    at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)

    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)

    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)

    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)

    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)

    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)

    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)

    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)

    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)

    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)

    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)

    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)

    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)

    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)

    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)

    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)

    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)

    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)

    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)

    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)

    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)

    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

    at org.eclipse.jetty.server.Server.handle(Server.java:499)

    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)

    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

    at java.lang.Thread.run(Unknown Source)

Caused by: java.nio.file.DirectoryNotEmptyException: E:\Jenkins\jobs\DataViz\jobs\DashboardWebApp\jobs\Dashboard-Deploy\branches\story-MS-box.gumkf1.ti-selection\builds\lastSuccessfulBuild

    at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)

    at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)

    at java.nio.file.Files.deleteIfExists(Unknown Source)

    at hudson.Util.tryOnceDeleteFile(Util.java:287)

    at hudson.Util.deleteFile(Util.java:243)

    ... 172 more

Could you help finding a way to clean that or provide an update to automatically clean ?
Thanks

Originally reported by splashnenen, imported from: A lot of errors at jenkins start since upgrades to new branch models : Failed to update org.jenkinsci.plugins.workflow.job.WorkflowJob / Unable to delete ... lastSuccessfulBuild
  • status: Resolved
  • priority: Major
  • resolution: Not A Defect
  • resolved: 2017-07-27T12:21:31+00:00
  • imported: 2022/01/10
timja commented 7 years ago

jglick:

You have a corrupt filesystem. lastSuccessfulBuild is supposed to be a symlink to a folder, not an actual folder. Perhaps you did a bad backup & restore?

There is a core RFE somewhere to get rid of these symlinks.

timja commented 7 years ago

stephenconnolly:

Smells like a side-effect of a bad restore from backup. No assertions to the contrary. Re-open if you can provide evidence showing how this is not a bad backup restore issue