search

jenkinsci / lockable-resources-plugin

Lock resources against concurrent use
https://plugins.jenkins.io/lockable-resources
MIT License
87 stars 183 forks source link

Deadlock on Jenkins controller while using lockable resources #607

Closed andreyakostovsap closed 8 months ago

andreyakostovsap commented 8 months ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.414.3 OS: Linux - 5.4.0-169-generic Java: 17.0.9 (OpenJDK 64-Bit Server VM) --- Parameterized-Remote-Trigger:3.2.0 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 authentication-tokens:1.53.v1c90fd9191a_b_ authorize-project:1.7.1 blueocean:1.27.9 blueocean-bitbucket-pipeline:1.27.9 blueocean-commons:1.27.9 blueocean-config:1.27.9 blueocean-core-js:1.27.9 blueocean-dashboard:1.27.9 blueocean-display-url:2.4.2 blueocean-events:1.27.9 blueocean-git-pipeline:1.27.9 blueocean-github-pipeline:1.27.9 blueocean-i18n:1.27.9 blueocean-jwt:1.27.9 blueocean-personalization:1.27.9 blueocean-pipeline-api-impl:1.27.9 blueocean-pipeline-editor:1.27.9 blueocean-pipeline-scm-api:1.27.9 blueocean-rest:1.27.9 blueocean-rest-impl:1.27.9 blueocean-web:1.27.9 bootstrap5-api:5.3.2-3 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1128.v717130d4f816 build-user-vars-plugin:1.9 build-with-parameters:76.v9382db_f78962 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.0.2 cloudbees-bitbucket-branch-source:856.v04c46c86f911 cloudbees-folder:6.858.v898218f3609d command-launcher:107.v773860566e2e commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-95.v22a_d30ee5d36 config-file-provider:959.vcff671a_4518b_ copyartifact:722.v0662a_9b_e22a_c credentials:1311.vcf0a_900b_37c2 credentials-binding:642.v737c34dea_6c2 data-tables-api:1.13.6-5 display-url-api:2.200.vb_9327d658781 docker-commons:439.va_3cb_0a_6a_fb_29 docker-workflow:572.v950f58993843 durable-task:543.v262f6a_803410 echarts-api:5.4.0-7 extended-read-permission:53.v6499940139e5 external-monitor-job:215.v2e88e894db_f8 favorite:2.208.v91d65b_7792a_c flaky-test-handler:1.2.3 font-awesome-api:6.5.1-1 gerrit-trigger:2.39.3 gerrit-trigger-manual-patch-set:0.6 git:5.2.1 git-client:4.6.0 github:1.37.3.1 github-api:1.318-461.v7a_c09c9fa_d63 github-branch-source:1767.va_7d01ea_c7256 gson-api:2.10.1-15.v0d99f670e0a_7 h2-api:11.1.4.199-12.v9f4244395f7a_ handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 htmlpublisher:1.32 http_request:1.18 influxdb:3.6 instance-identity:185.v303dc7c645f9 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.16.1-373.ve709c6871598 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:243.vb_b_503b_b_45537 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.9 jjwt-api:0.11.5-77.v646c772fddb_0 joda-time-api:2.12.5-5.v5495a_235fedf jquery3-api:3.7.1-1 jsch:0.2.16-86.v42e010d9484b_ json-path-api:2.8.0-21.v8b_7dc8b_1037b_ junit:1252.vfc2e5efa_294f kubernetes:4054.v2da_8e2794884 kubernetes-client-api:6.8.1-224.vd388fca_4db_3b_ kubernetes-credentials:0.11 lockable-resources:1222.v3d55a_36d63e4 mailer:463.vedf8358e006b_ mapdb-api:1.0.9-28.vf251ce40855d matrix-project:822.v01b_8c85d16d2 maven-plugin:3.23 metrics:4.2.18-442.v02e107157925 mina-sshd-api-common:2.11.0-86.v836f585d47fa_ mina-sshd-api-core:2.11.0-86.v836f585d47fa_ okhttp-api:4.11.0-157.v6852a_a_fa_ec11 pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:689.veec561a_dee13 pipeline-input-step:477.v339683a_8d55e pipeline-maven:1368.vfb_8509d7b_869 pipeline-maven-api:1368.vfb_8509d7b_869 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2150.v4cfd8916915c pipeline-model-definition:2.2150.v4cfd8916915c pipeline-model-extensions:2.2150.v4cfd8916915c pipeline-rest-api:2.34 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2150.v4cfd8916915c pipeline-stage-view:2.34 pipeline-utility-steps:2.16.0 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.8.0 pubsub-light:1.18 rebuild:330.v645b_7df10e2a_ resource-disposer:0.23 role-strategy:689.v731678c3e0eb_ saml:4.429.v9a_781a_61f1da_ scm-api:676.v886669a_199a_a_ scm-sync-configuration:0.0.10 script-security:1310.vf24a_dfce068b_ slack:684.v833089650554 snakeyaml-api:2.2-111.vc6598e30cc65 sonar:2.16.1 sse-gateway:1.26 ssh-agent:346.vda_a_c4f2c8e50 ssh-credentials:308.ve4497b_ccd8f4 sshd:3.312.v1c601b_c83b_0e structs:325.vcb_307d2a_2782 subversion:2.17.3 timestamper:1.26 token-macro:400.v35420b_922dcb_ trilead-api:2.133.vfb_8a_7b_9c5dd1 variant:60.v7290fc0eb_b_cd workflow-aggregator:596.v8c21c963d92d workflow-api:1283.v99c10937efcb_ workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3832.vc43e04d6d68c workflow-durable-task-step:1313.vcb_970b_d2a_fb_3 workflow-job:1385.vb_58b_86ea_fff1 workflow-multibranch:770.v1a_d0708dd1f6 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:865.v43e78cc44e0d ws-cleanup:0.45 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Debian 12 (in Docker)

Reproduction steps

Not sure how exactly it could be reproduced, but issue seems to be happening when we have ~10 pipelines running concurrently and utilizing lockable resources (not necessarily the same resources).

Expected Results

No deadlocks.

Actual Results

Running pipelines freeze and never finish. Triggering a thread dump of the Jenkins controller process outputs information that a deadlock was found.

Anything else?

Issue seems to be caused by the changes introduced in #586. The thread dump prints the following information (pipeline names are redacted):

Found one Java-level deadlock:
=============================
"Running CpsFlowExecution[Owner[tests.integration.0/34719:tests.integration.0 #34719]]":
  waiting to lock monitor 0x00007f19201bcf40 (object 0x000000060f6dbe10, a org.jenkins.plugins.lockableresources.LockableResourcesManager),
  which is held by "Running CpsFlowExecution[Owner[tests.integration.1/15001:tests.integration.1 #15001]]"

"Running CpsFlowExecution[Owner[tests.integration.1/15001:tests.integration.1 #15001]]":
  waiting to lock monitor 0x00007f1a3c3c4f40 (object 0x000000060d81d1d0, a java.lang.Object),
  which is held by "Running CpsFlowExecution[Owner[tests.integration.1/15002:tests.integration.1 #15002]]"

"Running CpsFlowExecution[Owner[tests.integration.1/15002:tests.integration.1 #15002]]":
  waiting to lock monitor 0x00007f19201bcf40 (object 0x000000060f6dbe10, a org.jenkins.plugins.lockableresources.LockableResourcesManager),
  which is held by "Running CpsFlowExecution[Owner[tests.integration.1/15001:tests.integration.1 #15001]]"

Java stack information for the threads listed above:
===================================================
"Running CpsFlowExecution[Owner[tests.integration.0/34719:tests.integration.0 #34719]]":
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:567)
    - waiting to lock <0x000000060f6dbe10> (a org.jenkins.plugins.lockableresources.LockableResourcesManager)
    at org.jenkins.plugins.lockableresources.LockStepExecution$Callback.finished(LockStepExecution.java:187)
    at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:118)
    at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:375)
    at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:70)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:149)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:146)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:423)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:331)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:295)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:97)
    at java.util.concurrent.FutureTask.run(java.base@17.0.9/FutureTask.java:264)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
    at jenkins.util.ErrorLoggingExecutorService$$Lambda$723/0x00007f19d0c75b88.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.9/Executors.java:539)
    at java.util.concurrent.FutureTask.run(java.base@17.0.9/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.9/ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.9/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(java.base@17.0.9/Thread.java:840)
"Running CpsFlowExecution[Owner[tests.integration.1/15001:tests.integration.1 #15001]]":
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.getResourcesWithLabel(LockableResourcesManager.java:227)
    - waiting to lock <0x000000060d81d1d0> (a java.lang.Object)
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.getResourcesWithLabel(LockableResourcesManager.java:216)
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.checkResourcesAvailability(LockableResourcesManager.java:1109)
    - locked <0x000000060f6dbe10> (a org.jenkins.plugins.lockableresources.LockableResourcesManager)
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.checkResourcesAvailability(LockableResourcesManager.java:1077)
    - locked <0x000000060f6dbe10> (a org.jenkins.plugins.lockableresources.LockableResourcesManager)
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.getNextQueuedContext(LockableResourcesManager.java:707)
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.getNextQueuedContext(LockableResourcesManager.java:682)
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:577)
    - locked <0x000000060f6dbe10> (a org.jenkins.plugins.lockableresources.LockableResourcesManager)
    at org.jenkins.plugins.lockableresources.LockStepExecution$Callback.finished(LockStepExecution.java:187)
    at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:118)
    at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:375)
    at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:70)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:149)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:146)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:423)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:331)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:295)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:97)
    at java.util.concurrent.FutureTask.run(java.base@17.0.9/FutureTask.java:264)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
    at jenkins.util.ErrorLoggingExecutorService$$Lambda$723/0x00007f19d0c75b88.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.9/Executors.java:539)
    at java.util.concurrent.FutureTask.run(java.base@17.0.9/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.9/ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.9/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(java.base@17.0.9/Thread.java:840)
"Running CpsFlowExecution[Owner[tests.integration.1/15002:tests.integration.1 #15002]]":
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.fromName(LockableResourcesManager.java:270)
    - waiting to lock <0x000000060f6dbe10> (a org.jenkins.plugins.lockableresources.LockableResourcesManager)
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.resourceExist(LockableResourcesManager.java:292)
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.addResource(LockableResourcesManager.java:803)
    - locked <0x000000060d81d1d0> (a java.lang.Object)
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.createResource(LockableResourcesManager.java:753)
    at org.jenkins.plugins.lockableresources.LockStepExecution.start(LockStepExecution.java:55)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:323)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:47)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
    at org.jenkinsci.plugins.workflow.cps.LoggingInvoker.methodCall(LoggingInvoker.java:105)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:90)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:116)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:85)
    at jdk.internal.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.9/DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(java.base@17.0.9/Method.java:568)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:152)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:146)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:423)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:331)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:295)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:97)
    at java.util.concurrent.FutureTask.run(java.base@17.0.9/FutureTask.java:264)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
    at jenkins.util.ErrorLoggingExecutorService$$Lambda$723/0x00007f19d0c75b88.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.9/Executors.java:539)
    at java.util.concurrent.FutureTask.run(java.base@17.0.9/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.9/ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.9/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(java.base@17.0.9/Thread.java:840)

Are you interested in contributing a fix?

No response

mPokornyETM commented 8 months ago

Duplicate #503

mPokornyETM commented 8 months ago

Thx for reporting issues with helpful stack. This is because, the synchronized are used in wrong way.

mPokornyETM commented 8 months ago

I need to merge my changes used by my own instance #558 into this master. But who will review 17 changed files? That the reason, why I split it in few smaller PRs. Let me check how can be merged it effectively, because there are many other things done.

fraz3alpha commented 8 months ago

In case it helps, we've experienced this quite a bit today after upgrading the plugin, so we've gone back to our previously installed version.

We've got a much more succinct deadlock with just two threads, where they are each waiting on each other, having apparently tried to take the same two locks in different orders (stacktraces truncated for clarity):

Thread 1:

        at org.jenkins.plugins.lockableresources.LockableResourcesManager.fromName(LockableResourcesManager.java:321)
    -  blocked on org.jenkins.plugins.lockableresources.LockableResourcesManager@c5ac193
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.resourceExist(LockableResourcesManager.java:343)
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.addResource(LockableResourcesManager.java:866)
    -  locked java.lang.Object@56c05066
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.createResource(LockableResourcesManager.java:816)

Thread 2:

    at org.jenkins.plugins.lockableresources.LockableResourcesManager.save(LockableResourcesManager.java:1451)
    -  blocked on java.lang.Object@56c05066
    at org.jenkins.plugins.lockableresources.LockableResourcesManager.lock(LockableResourcesManager.java:567)
    -  locked org.jenkins.plugins.lockableresources.LockableResourcesManager@c5ac193

Hopefully that is useful in validating your proposed changes.

mark-goodall commented 8 months ago

We have also been experiencing this issue over the last couple of days, which we incorrectly attributed to GitHub API issues. We have been experiencing deadlocks when acquiring locks and on restarting the Jenkins controller we get concurrent modification exceptions (solved by removing the plugin and reinstalling currently), which should have been fixed by 1218.va_3dd45e2b_fa_7?.

I can see that there have been a few releases recently which seem to be related to this issue. Is there a version which is safe to roll back to?

mPokornyETM commented 8 months ago

Official probably not. We use version builded in #558 Probably good candidate will be https://ci.jenkins.io/job/Plugins/job/lockable-resources-plugin/job/PR-558/40/artifact/

In that case you need to install plugin manually (just upload in the plugin page and reboot jenkins)

With that versions we runs over mounths without all the stucks and concurent modifications issues.

When somebody else try that, and confirm that it works, I will merge it as is. Otherwise it will takes time (currently I am only one maintaner here, and provide so deep and many changes without review, secondar test and discussions ist not the best idea)

mark-goodall commented 8 months ago

Thanks, I will update to that version overnight. Haven't seen the issue with https://github.com/jenkinsci/lockable-resources-plugin/releases/tag/1212.v1a_d65ecea_115 in the past 2 hours, previously had it occur twice in the last 6 hours with the latest release.

I had a brief look at the code (but unfortunately I would be unable to help with the workflow, I'm not experienced with Jenkins plugins) to determine if it was some kind of cache or database which could be cleared out to reduce the issue, as we have a lot of short lived locks, and it appears to be only storing locks which are in use. The https://Jenkins_URL/lockable-resources/ is rather helpful too

rhadjiyski commented 8 months ago

Last Version where still works is 1222.v3d55a_36d63e4 we had Deadlocks for the next two actual Versions

reftel commented 8 months ago

We have run into a very similar deadlock, where one thread runs freeResources (which is synchronized on the LockableResourcesManager instance), and calls save (which tries to also synchronized on syncResources), while another thread is running addResource and (while being synchronized on syncResources) calls fromName (which is synchronized on the LockableResourcesManager instance).

lmife commented 8 months ago

I'm also experiencing this issue. Rolled back to 1218.va_3dd45e2b_fa_7 (which had been the last version we had been using)

advorsky73 commented 8 months ago

As well experiencing the same lockups since last update on Saturday. Went back to [1222.v3d55a_36d63e4] as stated above and its working again!

mPokornyETM commented 8 months ago

Ok, finnaly i merged #558 and this issue shauld be fixed in the next release. I will keep it now opne, until somebody confirm, that the next release works as well. Thx

zeners commented 8 months ago

looks good to me: lockable resources-view works while resource is locked with 1232.v512d6c434eb_d