timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-60006] Cannot delete orphaned items in Multibranch pipeline #4662

Open timja opened 4 years ago

timja commented 4 years ago

I believe this is similar to JENKINS-58733 and JENKINS-58998 but it reproduces in 2.190.

The issue I'm facing is that Multibranch Scan cannot delete orphaned projects. When I try to delete them using Jenkins UI I get the same error. Deletion is blocked by a stash being ostensibly used by someone.

I don't have the necessary access to Jenkins to find who is using the file. I'm working on figuring that out.

Here is the Scan log output (the error from the UI matches):

Will remove PR-12345
FATAL: Failed to recompute children of ***PROJECT_NAME***
jenkins.util.io.CompositeIOException: Unable to delete '/var/jenkins_home/jobs/***PROJECT_NAME***/branches/PR-12345'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
    at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:99)
    at hudson.Util.deleteRecursive(Util.java:293)
    at hudson.Util.deleteRecursive(Util.java:282)
    at hudson.model.AbstractItem.performDelete(AbstractItem.java:801)
    at org.jenkinsci.plugins.workflow.job.WorkflowJob.performDelete(WorkflowJob.java:650)
    at hudson.model.AbstractItem.delete(AbstractItem.java:785)
    at hudson.model.Job.delete(Job.java:676)
    at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:290)
    at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
    at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1026)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
jenkins.util.io.CompositeIOException: Unable to remove file /var/jenkins_home/jobs/***PROJECT_NAME***/branches/PR-12345/builds/3/stashes/.nfs0000000095fe10b1000020ec
    at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:252)
    at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
    at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
    at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
    at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
    at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
    at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
    at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
    at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
    at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
    at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
    at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
    at hudson.Util.deleteRecursive(Util.java:293)
    ...

See the attached file for the full stack trace.

I marked this as Critical as it eventually will fill the disk space and block any other builds from running.


Originally reported by saynurin, imported from: Cannot delete orphaned items in Multibranch pipeline
  • assignee: igalg
  • status: Open
  • priority: Critical
  • resolution: Unresolved
  • imported: 2022/01/10
timja commented 2 years ago

warden:

 I can see similar issue on GH Action builds:

jenkins.util.io.CompositeIOException: Unable to delete 'D:\a\jira-plugin\jira-plugin\target\tmp\j h851780876193825225'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
143 at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:99)
144 at hudson.Util.deleteRecursive(Util.java:296)
145 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1398)
146 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1394)
147 at hudson.FilePath.act(FilePath.java:1164)
148 at hudson.FilePath.act(FilePath.java:1147)
149 at hudson.FilePath.deleteRecursive(FilePath.java:1392)
150 at org.jvnet.hudson.test.TemporaryDirectoryAllocator.dispose(TemporaryDirectoryAllocator.java:93)
151 at org.jvnet.hudson.test.TestEnvironment.dispose(TestEnvironment.java:84)
152 at org.jvnet.hudson.test.JenkinsRule.after(JenkinsRule.java:513)
153 at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:619)
154 at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
155 at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
156 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
157 at java.base/java.lang.Thread.run(Thread.java:829)
158 Suppressed: java.nio.file.AccessDeniedException: D:\a\jira-plugin\jira-plugin\target\tmp\j h851780876193825225\jobs\test0\builds\1\log
159     at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
160     at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
161     at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
162     at java.base/sun.nio.fs.WindowsFileAttributeViews$Dos.updateAttributes(WindowsFileAttributeViews.java:242)
163     at java.base/sun.nio.fs.WindowsFileAttributeViews$Dos.setReadOnly(WindowsFileAttributeViews.java:248)
164     at jenkins.util.io.PathRemover.makeWritable(PathRemover.java:298)
165     at jenkins.util.io.PathRemover.makeRemovable(PathRemover.java:259)
166     at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:239)
167     at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
168     at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
169     at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
170     at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
171     at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
172     at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
173     at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
174     at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
175     at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
176     at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
177     at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
178     at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
179     at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
180     ... 14 more