search

timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-46738] Self-deadlock in ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt #9360

Closed timja closed 7 years ago

timja commented 7 years ago

What is the business impact?

Jenkins stop working due the deadlock between support-core plugin and core/org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt(ExecutorStepExecution.java:570)

What is the incorrect behaviour?

The deadlock should not happen

What is the expected behaviour?

You should be able to automatically create a bundle and stop a build execution

Step by step to reproduce

N/A

Workaround

Restart the instance, disable support-core plugin or avoid to stop builds. Issue does not seems to be very frequent.

Regression

I don't think this is a regression.

Environment

support-core:2.32:not-pinned
Version: `2.7.21.1 (CloudBees Jenkins Enterprise 2.7.21.1-rolling)`

Deadlock stacktrace

Found one Java-level deadlock:
=============================
"SupportPlugin periodic bundle generator: writing support_2017-09-07_20.55.37.zip since Thu Sep 07 16:55:37 EDT 2017":
  waiting for ownable synchronizer 0x00000006f2ccb9e0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "Handling POST /computer/(master)/executors/0/stop from XXXXXX : RequestHandlerThread[#155]"
"Handling POST /computer/(master)/executors/0/stop from XXXXXX : RequestHandlerThread[#155]":
  waiting for ownable synchronizer 0x00000006f3362550, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "Handling POST /computer/(master)/executors/4/stop from XXXXX : RequestHandlerThread[#161]"
"Handling POST /computer/(master)/executors/4/stop from XXXXXX : RequestHandlerThread[#161]":
  waiting for ownable synchronizer 0x00000006f41de5e0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "Handling POST /computer/(master)/executors/6/stop from XXXXXXX : RequestHandlerThread[#162]"
"Handling POST /computer/(master)/executors/6/stop from XXXXXXXX : RequestHandlerThread[#162]":
  waiting for ownable synchronizer 0x00000006f2ccb9e0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "Handling POST /computer/(master)/executors/0/stop from XXXXXXX : RequestHandlerThread[#155]"

Found 1 deadlock.
Originally reported by fbelzunc, imported from: Self-deadlock in ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt
  • assignee: jglick
  • status: Closed
  • priority: Major
  • resolution: Fixed
  • resolved: 2017-09-27T14:39:54+00:00
  • imported: 2022/01/10
timja commented 7 years ago

jglick:

The support-core thread is innocent. We have three threads in a cyclic deadlock:

"Handling POST /computer/(master)/executors/…/stop …":
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <…> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
    at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
    at hudson.model.Executor.getCurrentExecutable(Executor.java:503)
    at hudson.model.Executor.of(Executor.java:966)
    at hudson.model.Run.getExecutor(Run.java:517)
    at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt(ExecutorStepExecution.java:570)
    at hudson.model.Executor.interrupt(Executor.java:227)
    at hudson.model.Executor.interrupt(Executor.java:197)
    at hudson.model.Executor.interrupt(Executor.java:187)
    at hudson.model.Executor.interrupt(Executor.java:173)
    at hudson.model.Executor.doStop(Executor.java:864)

Executor.interrupt acquires a per-Executor lock, then calls into foreign code (AsynchronousExecution.interrupt), which in this case is calling Executor.of, attempting to acquire the locks of other executors.

timja commented 7 years ago

svanoort:

Released with v2.15