[JENKINS-64766] Build delete fails via API when using NFS

[timja]

Deleting builds from the GUI has always worked. We have hundreds of unstable/failed builds that we would like to delete and are trying to do so via the API. When we do, the delete fails due to some NFS file locks. Additionally, for the delete that fail, the GUI will then also fail to delete that specific and many other builds until the entire jenkins service is restarted.

We are using the "doDelete" command, but not sure if there is something more close to the GUI's method that will avoid this.

Here is a basic curl command to delete build 3469 of "job_name" (obviously obfuscated), and the resulting error.

 

$ curl -X POST "http://username:apitoken@jenkins.example.com/job/job_name/3469/doDelete" -v
* About to connect() to jenkins.example.com port 80 (#0)
*   Trying 172.16.240.65...
* Connected to jenkins.example.com (172.16.240.65) port 80 (#0)
* Server auth using Basic with user 'username'
> POST /job/job_name/3469/doDelete HTTP/1.1
> Authorization: Basic Y3RvYmV5OjExOWZhYjBmZjJjMDQzNjVkODEyNWZlOTkzNmNlNjQxYjY=
> User-Agent: curl/7.29.0
> Host: jenkins.example.com
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.12.2
< Date: Sun, 31 Jan 2021 08:06:16 GMT
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Content-Type-Options: nosniff
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: no-cache,no-store,must-revalidate
< X-Hudson-Theme: default
< Referrer-Policy: same-origin
< Cross-Origin-Opener-Policy: same-origin
< Set-Cookie: JSESSIONID.f99c08f6=node092f94enswdpnbcww0shbx6oc810.node0; Path=/; HttpOnly
< X-Hudson: 1.395
< X-Jenkins: 2.263.2
< X-Jenkins-Session: 99c224a5
< X-Frame-Options: sameorigin
< X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAk1rU873dexIkg2/yWbAUT4eRcbD+0RWrJu2Hku721SR8WPGSBdrWPU3/bHPOAbAuP+rX8Vlbxz6+QjY8eep9VOZnUtWVa/+lTJC158T8iiffpezVprmNfN4RGFc0DUFsTsvZ/3Zxwx+gtjZ7/LRFN77LDmMT8cxtw9L0rQlRt4N6AgWpta42uPgP7vYtAM9AWJ3gfgHlrGjcEKRjDlbqf6Ad+7rWwyyiX7S9L/io2vo9AA9iG6siccHOvvZ8dM8sh3j7FF0D9kaQAdkiH+jj8ysEVEth9DZTbj36Ly28C7MmRkUBR/XnRbYq8wXGpCkpkjlKcVqrIC8d4xuSlHt8gQIDAQAB
<   
...
"post" action="doDelete">
"error">Deletion  of the build failed
"now" type="submit" value="Retry delete" class="submit-button primary ">
"document.getElementById('delete-error').style.display='block';return false" href="">Show reason
" style="display: none">
Also:       Also:   java.nio.file.FileSystemException: /var/lib/jenkins/jobs/job_name/builds/.3469/archive/zynq/build_1rx1tx_south/.nfs000000000d0df5930000002d: Device or resource busy
            at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
            at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
            at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
            at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
            at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
            at java.nio.file.Files.deleteIfExists(Files.java:1165)
            at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
    Also:   java.nio.file.FileSystemException: /var/lib/jenkins/jobs/job_name/builds/.3469/archive/zynq/build_1rx1tx_south/.nfs000000000d0df5930000002d: Device or resource busy
            at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
            at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
            at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
            at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
            at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
            at java.nio.file.Files.deleteIfExists(Files.java:1165)
            at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove file /var/lib/jenkins/jobs/job_name/builds/.3469/archive/zynq/build_1rx1tx_south/.nfs000000000d0df5930000002d
        at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:252)
        at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
        at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
        at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
        at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
        at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
        at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also:       Also:   java.nio.file.DirectoryNotEmptyException: /var/lib/jenkins/jobs/job_name/builds/.3469/archive/zynq/build_1rx1tx_south
            at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
            at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
            at java.nio.file.Files.deleteIfExists(Files.java:1165)
            at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
    Also:   java.nio.file.DirectoryNotEmptyException: /var/lib/jenkins/jobs/job_name/builds/.3469/archive/zynq/build_1rx1tx_south
            at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
            at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
            at java.nio.file.Files.deleteIfExists(Files.java:1165)
            at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove directory /var/lib/jenkins/jobs/job_name/builds/.3469/archive/zynq/build_1rx1tx_south with directory contents: [/var/lib/jenkins/jobs/job_name/builds/.3469/archive/zynq/build_1rx1tx_south/.nfs000000000d0df5930000002d]
        at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:250)
        at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
        at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
        at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
        at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
        at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also:       Also:   java.nio.file.DirectoryNotEmptyException: /var/lib/jenkins/jobs/job_name/builds/.3469/archive/zynq
            at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
            at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
            at java.nio.file.Files.deleteIfExists(Files.java:1165)
            at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
    Also:   java.nio.file.DirectoryNotEmptyException: /var/lib/jenkins/jobs/job_name/builds/.3469/archive/zynq
            at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
            at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
            at java.nio.file.Files.deleteIfExists(Files.java:1165)
            at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove directory /var/lib/jenkins/jobs/job_name/builds/.3469/archive/zynq with directory contents: [/var/lib/jenkins/jobs/job_name/builds/.3469/archive/zynq/build_1rx1tx_south]
        at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:250)
        at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
        at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
        at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
        at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also:       Also:   java.nio.file.DirectoryNotEmptyException: /var/lib/jenkins/jobs/job_name/builds/.3469/archive
            at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
            at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
            at java.nio.file.Files.deleteIfExists(Files.java:1165)
            at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
    Also:   java.nio.file.DirectoryNotEmptyException: /var/lib/jenkins/jobs/job_name/builds/.3469/archive
            at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
            at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
            at java.nio.file.Files.deleteIfExists(Files.java:1165)
            at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove directory /var/lib/jenkins/jobs/job_name/builds/.3469/archive with directory contents: [/var/lib/jenkins/jobs/job_name/builds/.3469/archive/zynq]
        at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:250)
        at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
        at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
        at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
Also:       Also:   java.nio.file.DirectoryNotEmptyException: /var/lib/jenkins/jobs/job_name/builds/.3469
            at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
            at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
            at java.nio.file.Files.deleteIfExists(Files.java:1165)
            at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
    Also:   java.nio.file.DirectoryNotEmptyException: /var/lib/jenkins/jobs/job_name/builds/.3469
            at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
            at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
            at java.nio.file.Files.deleteIfExists(Files.java:1165)
            at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:241)
jenkins.util.io.CompositeIOException: Unable to remove directory /var/lib/jenkins/jobs/job_name/builds/.3469 with directory contents: [/var/lib/jenkins/jobs/job_name/builds/.3469/archive]
        at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:250)
        at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
        at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
        at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
jenkins.util.io.CompositeIOException: Unable to delete '/var/lib/jenkins/jobs/job_name/builds/.3469'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
    at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:99)
    at hudson.Util.deleteRecursive(Util.java:296)
    at hudson.Util.deleteRecursive(Util.java:285)
    at hudson.model.Run.delete(Run.java:1681)
    at hudson.model.Run.doDoDelete(Run.java:2369)
    at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
    at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
    at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
    at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
    at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:536)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1633)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at org.jenkinsci.plugins.modernstatus.ModernStatusFilter.doFilter(ModernStatusFilter.java:52)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
    at jenkins.security.ApiCrumbExclusion.process(ApiCrumbExclusion.java:48)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:127)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:135)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:82)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
    at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
    at java.lang.Thread.run(Thread.java:748)

"page-footer">

"container-fluid">

"page-footer__flex-row">

"page-footer__footer-id-placeholder" id="footer">

'stylesheet' href='/adjuncts/99c224a5/jenkins/management/AdministrativeMonitorsDecorator/resources.css' type='text/css' />

"visible-am-container">"toggleVisibleAmList(event)" id="visible-am-button" href="#" title="There are 3 active administrative monitors.">"" focusable="false" class="svg-icon ">static/99c224a5/images/material-icons/svg-sprite-social-symbol.svg#ic_notifications_24px">">3

">

">

">

The following installed plugins are deprecated

" href="https://issues.jenkins-ci.org/browse/INFRA-2487" target="_blank">Static Analysis Utilities

" href="https://issues.jenkins-ci.org/browse/INFRA-2487" target="_blank">Continuous Integration game

" href="https://plugins.jenkins.io/pipeline-model-declarative-agent" target="_blank">Pipeline: Declarative Agent API

In general, this means that these plugins are either obsolete, no longer being developed, or may no longer work.
See the linked web pages for further information about the cause for the deprecation, and suggestions on how to proceed.

">

New version of Jenkins (2.263.3) is available for "https://updates.jenkins.io/download/war/2.263.3/jenkins.war">download ("https://jenkins.io/changelog-stable">changelog).

"am-message">

"alert alert-danger" role="alert">

"post" action="/administrativeMonitor/jenkins.security.UpdateSiteWarningsMonitor/forward">"fix" type="submit" value="Go to plugin manager" class="submit-button primary ">"configure" type="submit" value="Configure which of these warnings are shown" class="submit-button primary ">

Warnings have been published for the following currently installed components:

Jenkins 2.263.2 core and libraries

"https://www.jenkins.io/security/advisory/2021-01-26/" rel="noopener noreferrer" target="_blank">Arbitrary file read vulnerability in workspace browsers

"http://wiki.jenkins-ci.org/x/CwDgAQ" rel="noopener noreferrer" target="_blank">Static Analysis Utilities 1.96

"https://www.jenkins.io/security/advisory/2020-11-04/#SECURITY-1907" rel="noopener noreferrer" target="_blank">Stored XSS vulnerability

"https://plugins.jenkins.io/project-inheritance" rel="noopener noreferrer" target="_blank">Project Inheritance Plugin 19.08.02

"https://jenkins.io/security/advisory/2020-06-03/#SECURITY-1582" rel="noopener noreferrer" target="_blank">Missing permission check

Additional warnings are hidden due to the current security configuration.

"text-align: center; margin: 10px 0 0 0;">"document.location.href='/manage';" href="/manage">Manage Jenkins

"page-footer__links rest_api hidden-xs">"api/">REST API

"page-footer__links page-footer__links--white jenkins_ver">"noopener noreferrer" href="https://jenkins.io/" target="_blank">Jenki* Connection #0 to host jenkins.example.com left intact ns 2.263.2

[timja]

markewaite:

The simplest advice is to not use NFS mounts for Jenkins data directories. If you cannot avoid using NFS to mount a Jenkins data directory, refer to https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide for additional hints.

[timja]

tobeychris:

Sadly that is not an option for us - we have 1000+ jobs each with hundreds of builds with very large artifacts currently using 45 TB of storage.

We use a TrueNAS fileserver to serve Jenkins with a dedicated pool. It is currently using relatime and NFS v3.

Maybe someone smarter than me can see if there is anything bad here:

$ iostat -x
Linux 3.10.0-1062.9.1.el7.x86_64 (jenkins)  02/04/2021  _x86_64_    (8 CPU)avg-cpu:  %user   %nice %system %iowait  %steal   %idle
   0.42    0.00    0.28    0.09    0.00   99.21Device: rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda       0.00     0.08    0.06    1.58     2.67    35.29    46.53     0.01    4.18    0.91    4.30   0.38   0.06
dm-0      0.00     0.00    0.06    1.65     2.58    35.28    44.34     0.01    4.14    0.90    4.25   0.37   0.06
dm-1      0.00     0.00    0.00    0.00     0.01     0.00    50.09     0.00    0.17    0.17    0.00   0.12   0.00

/import/jenkins from truenas:/mnt/SG2/jenkins
 Flags: rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.240.35,mountvers=3,mountport=794,mountproto=udp,local_lock=none,addr=172.16.240.35

$  nfsstat -rc
Client rpc stats:
calls      retrans    authrefrsh
39031121   0  39035440

[timja]

markewaite:

The guidance from the CloudBees page recommends NFS v4.1

[timja]

tobeychris:

Are the internal flows for deletes via the GUI button and the API /doDelete significantly different?

[timja]

markewaite:

I would expect the internal flow for the GUI button and the doDelete API to be the same.ll