timja
commented
5 years ago I received a replay of the problem, i have attached full jstack
Linked log
root@sbt-oainr-0040:/u01/haproxy$ cat haproxy.log | grep 'efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check'10.247.116.176:52805 [02/Jul/2019:09:19:53.053] jenkins_frontend~ jenkinsci_efs-backend/jenkins-master-efs-002.vm.mos.cloud.sbrf.ru 495/0/1/4/581 302 258 - - ---- 533/338/3/3/0 0/0 "GET /efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check/sbt%24ThroughTheEFS HTTP/1.1"10.247.116.176:52807 [02/Jul/2019:09:19:53.611] jenkins_frontend~ jenkinsci_efs-backend/jenkins-master-efs-002.vm.mos.cloud.sbrf.ru 67/0/0/-1/375426 400 187 - - CH-- 589/388/41/9/0 0/0 "GET /efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check/sbt%24ThroughTheEFS/ HTTP/1.1"
root@jenkins-master-efs-002:~$ cat /opt/jenkins/log/jenkins.log | grep 'efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check'WARNING: Error while serving http://sbt-jenkins.ca.sbrf.ru/efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check/sbt%24ThroughTheEFS/14973//api/jsonWARNING: Error while serving http://sbt-jenkins.ca.sbrf.ru/efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check/sbt%24ThroughTheEFS/14974//api/jsonWARNING: Error while serving http://sbt-jenkins.ca.sbrf.ru/efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check/sbt%24ThroughTheEFS/14975//api/jsonWARNING: Error while serving http://sbt-jenkins.ca.sbrf.ru/efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check/sbt%24ThroughTheEFS/14976//api/jsonWARNING: Error while serving http://sbt-jenkins.ca.sbrf.ru/efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check/sbt%24ThroughTheEFS/14977//api/jsonINFO: Export build info to Confluence: {"buildKey":"51837e38-e3dd-4b06-8fc0-4910184b6812","buildName":"14979","buildUrl":"https://sbt-jenkins.ca.sbrf.ru/efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check/14979/","buildSuccess":"false","startDate":"1562059901","failTestCount":"2","skipTestCount":"0","totalTestCount":"16","buildTime":"1562049101390","buildStatus":"UNSTABLE"}INFO: Export build info to Confluence: {"buildKey":"51837e38-e3dd-4b06-8fc0-4910184b6812","buildName":"14980","buildUrl":"https://sbt-jenkins.ca.sbrf.ru/efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check/14980/","buildSuccess":"false","startDate":"1562060273","failTestCount":"1","skipTestCount":"0","totalTestCount":"16","buildTime":"1562049473822","buildStatus":"UNSTABLE"}INFO: Export build info to Confluence: {"buildKey":"51837e38-e3dd-4b06-8fc0-4910184b6812","buildName":"14981","buildUrl":"https://sbt-jenkins.ca.sbrf.ru/efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check/14981/","buildSuccess":"false","startDate":"1562060640","failTestCount":"1","skipTestCount":"0","totalTestCount":"16","buildTime":"1562049840735","buildStatus":"UNSTABLE"}INFO: Export build info to Confluence: {"buildKey":"51837e38-e3dd-4b06-8fc0-4910184b6812","buildName":"14982","buildUrl":"https://sbt-jenkins.ca.sbrf.ru/efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check/14982/","buildSuccess":"false","startDate":"1562061240","failTestCount":"1","skipTestCount":"0","totalTestCount":"16","buildTime":"1562050440736","buildStatus":"UNSTABLE"}INFO: Export build info to Confluence: {"buildKey":"51837e38-e3dd-4b06-8fc0-4910184b6812","buildName":"14983","buildUrl":"https://sbt-jenkins.ca.sbrf.ru/efs/job/UFS/job/DevOps/job/Cards/job/ift-health-check/14983/","buildSuccess":"false","startDate":"1562061840","failTestCount":"1","skipTestCount":"0","totalTestCount":"16","buildTime":"1562051040747","buildStatus":"UNSTABLE"}
On jenkins-2.164.2 i take deadlocks. A similar request was resolved earlier https://issues.jenkins-ci.org/browse/JENKINS-55361
It's resource 0x00000002fa1afc88 wait all Util.timer which does not allow the executor to be released and to launch new tasks
Jstack blocker thread in file
Originally reported by vnenashev, imported from: deadlocks ON build api/json