[JENKINS-50663] Deadlock on jenkins startup

[timja]

After plugin update jenkins hangs on startup. Attached thread dump showing deadlock. Here's short version:

Found one Java-level deadlock:
=============================
"CustomOrganizationFolderDescriptor.addSpecificDescriptors":
  waiting to lock monitor 0x00007f5510004f38 (object 0x00000006c6654678, a hudson.ExtensionList$Lock),
  which is held by "GitSCM.onLoaded"
"GitSCM.onLoaded":
  waiting for ownable synchronizer 0x00000006c6eb7488, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "SonarGlobalConfiguration.migrate"
"SonarGlobalConfiguration.migrate":
  waiting to lock monitor 0x00007f5510004f38 (object 0x00000006c6654678, a hudson.ExtensionList$Lock),
  which is held by "GitSCM.onLoaded"

---

Originally reported by jchorko, imported from: Deadlock on jenkins startup

• status: Resolved
• priority: Blocker
• resolution: Fixed
• resolved: 2019-02-04T14:11:48+00:00
• imported: 2022/01/10

[timja]

edtech:

I had the same problem. It's the new Sonar 2.7 plugin that creates the problem. I restored the 2.6.1 and Jenkins started properly.

[timja]

martinmajewski:

How to revert back to an older version, when Jenkins is not starting?

 

[timja]

giannello:

Same here.

martinmajewski go to the jenkins data directory, then `plugins`, you'll find `sonar.bak` and `sonar.jpi`. Remove the latter, and rename the former to `sonar.jpi`. Restart jenkins and you're good to go

[timja]

svanoort:

I have upgraded this to Blocker because it prevents Jenkins from functioning at all. Note that I'm not impacted, but am something of collateral damage since plugins I maintain caught the initial blame until we investigated.

[timja]

kpop:

Same here, workaround by giannello allowed Jenkins to start again.

[timja]

cobexer:

FYI: Apparently a fixed Sonar plugin is already published: https://plugins.jenkins.io/sonar?why-is-this-caching-so-damn-bad

Fix: https://github.com/jenkinsci/sonarqube-plugin/commit/735513927f6b3811cf9ced1a38681913639b6ab0

 

[timja]

henryju:

Fixed in 2.7.1

Ticket on our side: https://jira.sonarsource.com/browse/SONARJNKNS-294

[timja]

kpop:

On my setup, Sonar 2.7.1 fixes the startup issue. Thank you henryju for the quick fix!

[timja]

jglick:

While a Sonar plugin change may have triggered this bug, it looks more like a core issue to me. The deadlock is between GuiceFinder and ExtensionList, but possibly OldDataMonitor.report should not be acquiring a lock either since it is involved at a low level in initialization—perhaps it could queue up reports until after startup has completed, or at least extensions have been prepared, or perhaps data could become a static WeakHashMap reference.

[timja]

owenmehegan:

henryju please consider submitting a request to remove 2.7.0 from the update center, given that installing it can render Jenkins unable to start at all. See https://jenkins.io/doc/developer/publishing/removing-from-distribution/

[timja]

scm_issue_link:

Code changed in jenkins
User: Oleg Nenashev
Path:
src/test/java/org/jenkinsci/plugins/pipeline/utility/steps/conf/mf/ReadManifestStepTest.java
http://jenkins-ci.org/commit/pipeline-utility-steps-plugin/982516b2fc8c2f641fb362da915f6bb9a037d76b
Log:
JENKINS-50663 - Restore Remoting manifest checks as requested by @rsandell

[timja]

dlouzado:

In my case, updating to version 2.7.1 did not fix the problem.

2.7.1 did work on a new installation and did not on my staging environment. Jenkins startup freezed again and no logs were written. Cannot use this version on production env.

[timja]

oleg_nenashev:

 

JENKINS-55361 should fix it in 2.163 from what I see

[timja]

[Originally related to: JENKINS-50671]

[timja]

[Originally related to: JENKINS-55361]