timja
commented
4 years ago Hi thanks for the report!
> A possible cause of this might be {{AsyncResourceDisposer.worker}}s that run into errors due to insufficient file permissions for the user (jenkins) that runs the Jenkins daemon. It seems that these workers are unable to recover and skip the offending files and directories.
It is plausible, but it needs more investigation and analysis. One particular stack trace might be a red herring. If you have a test instance where you could reproduce the behavior, connecting a profiler would be a good step to collect the details (e.g. Java Flight Recorded). Tools like jvmtop or Monitoring plugin could also help to collect some per-thread diagnostics information from the instance.
> As a sidenote, I did not find the error in Jenkin's log files, which made it difficult to track down possible reasons
it is possible to go through the code in the stacktrace and discover what exactly suppresses it
Jenkins consumes vast amounts of CPU resources. E.g. on one machine during 103 days uptime htop reports 7387 CPU hours, i.e. in the mean 3 out of 8 cores are 100% utilzed during uptime. Actually it seems to be worse:
1 [|||||||||||||||||||||||||||||||||||||||||||| 71.7%] 5 [||||||||||||||||||||||||||||||||||||||||||||||||||| 84.0%] 2 [||||||||||||||||||||||||||||||||||||||||||||||||| 79.6%] 6 [||||||||||||||||||||||||||||||||||||||||||| 69.4%] 3 [|||||||||||||||||||||||||||||||||||||||||||||||76.6%] 7 [|||||||||||||||||||||||||||||||||||||||||||||||||||||| 88.7%] 4 [|||||||||||||||||||||||||||||||||||||||||||||||||||| 85.2%] 8 [||||||||||||||||||||||||||||||||||||||||||||||||||| 82.4%] Mem[|||||||||||||||||||||||||||||||||||||||||||||||||4.03G/7.79G] Tasks: 48; 6 running Swp[| 253M/16.0G] Load average: 8.30 8.02 8.28 Uptime: 103 days(!), 20:54:02 603 jenkins 20 0 7779M 2524M 5564 S 421. 31.6 7387h ├─ /java/jdk8/bin/java -XX:ErrorFile=/.mntpts/sdb1/jenkins/hs_err_pidjenkins...A possible cause of this might be {{AsyncResourceDisposer.worker}}s that run into errors due to insufficient file permissions for the user (jenkins) that runs the Jenkins daemon. It seems that these workers are unable to recover and skip the offending files and directories.
In our specific case the files have been created by Docker containers, but it may have other causes as well, so it's probably a quite common source of problems.
As a sidenote, I did not find the error in Jenkin's log files, which made it difficult to track down possible reasons for the bad Jenkins performance and Jobs that failed for no good reasons.
Originally reported by alexanderv, imported from: Extreme CPU utilization leading to bad performance and failing jobs