[JENKINS-49386] Windows Jenkins process locks the Log folder. Can't delete builds

[timja]

Running Jenkins as a scheduled task on Windows 2008 server. Recently upgraded to Jenkins v2.100 and since then I have problems deleting builds. Getting 'build in use error'

Checking for open file handles on the Windows server confirms that Jenkins is keeping open file handles on the log folders

This is true for several jobs. Some common plugins in use

[conditional-buildstep@1.3.5]
[groovy@1.30]
[envinject@1.93.1]
[scriptler@2.9]
[ssh@2.5]

---

Originally reported by ioannis, imported from: Windows Jenkins process locks the Log folder. Can't delete builds

• status: Open
• priority: Minor
• resolution: Unresolved
• imported: 2022/01/10

[timja]

oleg_nenashev:

Please provide a dump from http://file-leak-detector.kohsuke.org/

[timja]

ioannis:

Unfortunately, I can't get this to run successfully on my Windows server. Trying to bind to the Windows Jenkins PID I get and error

com.sun.tools.attach.AttachNotSupportedException: Insufficient memory or insufficient privileges to attach

Also trying to run with the http option I get no errors but the JVM starts and then immediately dies. Anyway, will this give me more information than the Windows Process Explorer? For example if some of my embedded groovy code or scriptlets are leaving the file handles open will I be able to detect it with the file-leak-detector? Thanks for the feedback

[timja]

kevin_randino:

I'd like to confirm that I am having the same issues as reported by Ioannis on Jenkins 2.107.2.  This is causing a major problem for us as our artifacts are not getting cleaned up do to the job being "in use" when the LogRotator runs and we are constantly running out of disk space because of it.  When the LogRotator runs I get the following error in my log:

 

Failed to rotate log java.io.IOException: M:\jenkins-jobs\Build-CheckoutInstall\builds\2415 is in use at hudson.model.Run.delete(Run.java:1505) at hudson.tasks.LogRotator.perform(LogRotator.java:145) at hudson.model.Job.logRotate(Job.java:472) at hudson.model.Run.execute(Run.java:1791) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429)

 

This does not appear to happen to all of our projects, but I can't find the common thread between the jobs it is happening on. It also does not appear to hold logs from builds that fail, only builds that succeed.  I attempted to run the file leak detector Oleg provided, but I am getting the same error that Ioannis is.  This is nearly a show stopper for us, as our Jenkins system runs out of disk space constantly while we can't clean artifacts.

Attached is a screenshot of some of the very many log files being held by Jenkins right now.

If there's anything I can do to help track this down, or if you can tell me what I'm doing wrong with the file leak detector, please let me know!

[timja]

kevin_randino:

Good news!  I figured out the issue on my end, and it was all of my own doing.  In the past when we used Jenkins 2.32.3, we never encountered this issue, it only started when we upgrade to v2.107.2.  It turns out we had a job that would open the log file of other jobs using the following groovy code:

reader = new BufferedReader(build.getLogReader())

The issue was we never did a reader.close() in the code, causing a handler for the log file to never be released.  The thing I find odd is this problem never popped up with Jenkins 2.32.3, so I still do believe there's some sort of change going on here, but knowing that this is not directly built into Jenkins, but rather something we did to ourselves is excellent news.  I've since started closing the BufferedReader, and now it looks like the handles are not being removed.  Sorry for the confusion on this one, but I hope this has helped others!

ioannis, I've got a bit of code for you that you can use to try to find any jobs that may be using a log reader like I was.  If you go to your http:///script web page, you can put this in there and it will scan all your jobs for the word "getLogReader" it may help you track down the same thing if it's happening on your system.  From there you just have to make sure they're getting closed.  Good luck!

// Find a specific string within all jenkins jobs.

search = "getLogReader()"

jenkins = Jenkins.instance.getInstance()

jenkins.getItems().each()
{
  if (it.getConfigFile().asString().contains(search))
  {
    println(jenkins.getRootUrl() + "job/" + it.getDisplayName() + "/configure")
  }
}