jenkinsci / lockable-resources-plugin

Lock resources against concurrent use
https://plugins.jenkins.io/lockable-resources
MIT License
86 stars 183 forks source link

Sometimes the plugin throws an IndexOutOfBoundsException with multiple locks #662

Open deiga opened 3 months ago

deiga commented 3 months ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.426.3 OS: Linux - 5.10.210-201.852.amzn2.x86_64 Java: 17.0.9 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- analysis-model-api:12.1.0 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 asm-api:9.7-33.v4d23ef79fcc8 authentication-tokens:1.53.v1c90fd9191a_b_ aws-credentials:231.v08a_59f17d742 aws-java-sdk:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-api-gateway:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-autoscaling:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-cloudformation:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-cloudfront:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-codebuild:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-codedeploy:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-ec2:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-ecr:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-ecs:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-efs:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-elasticbeanstalk:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-elasticloadbalancingv2:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-iam:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-kinesis:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-lambda:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-logs:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-minimal:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-organizations:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-secretsmanager:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-sns:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-sqs:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-ssm:1.12.696-451.v0651a_da_9ca_ec blueocean:1.27.12 blueocean-bitbucket-pipeline:1.27.12 blueocean-commons:1.27.12 blueocean-config:1.27.12 blueocean-core-js:1.27.12 blueocean-dashboard:1.27.12 blueocean-display-url:2.4.2 blueocean-events:1.27.12 blueocean-git-pipeline:1.27.12 blueocean-github-pipeline:1.27.12 blueocean-i18n:1.27.12 blueocean-jwt:1.27.12 blueocean-personalization:1.27.12 blueocean-pipeline-api-impl:1.27.12 blueocean-pipeline-editor:1.27.12 blueocean-pipeline-scm-api:1.27.12 blueocean-rest:1.27.12 blueocean-rest-impl:1.27.12 blueocean-web:1.27.12 bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1152.v6f101e97dd77 build-history-metrics-plugin:112.v476124de7dfc build-user-vars-plugin:1.9 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.2.0 cloudbees-bitbucket-branch-source:883.v041fa_695e9c2 cloudbees-disk-usage-simple:203.v3f46a_7462b_1a_ cloudbees-folder:6.858.v898218f3609d clover:4.14.2.596.vb_4d6475e990b_ command-launcher:107.v773860566e2e commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-95.v22a_d30ee5d36 conditional-buildstep:1.4.3 config-file-provider:968.ve1ca_eb_913f8c configuration-as-code:1775.v810dc950b_514 copyartifact:722.v0662a_9b_e22a_c credentials:1337.v60b_d7b_c7b_c9f credentials-binding:657.v2b_19db_7d6e6d dark-theme:439.vdef09f81f85e dashboard-view:2.508.va_74654f026d1 data-tables-api:2.0.3-1 datadog:6.0.3 dependency-check-jenkins-plugin:5.5.0 display-url-api:2.200.vb_9327d658781 docker-commons:439.va_3cb_0a_6a_fb_29 docker-workflow:572.v950f58993843 durable-task:550.v0930093c4b_a_6 echarts-api:5.5.0-1 extended-choice-parameter:381.v360a_25ea_017c extended-read-permission:53.v6499940139e5 extensible-choice-parameter:1.8.1 external-monitor-job:215.v2e88e894db_f8 favorite:2.208.v91d65b_7792a_c file-parameters:316.va_83a_1221db_a_7 font-awesome-api:6.5.1-3 forensics-api:2.4.0 git:5.2.1 git-client:4.7.0 git-forensics:2.1.0 git-prebuildmerge-trait:1.0 github:1.38.0 github-api:1.318-461.v7a_c09c9fa_d63 github-branch-source:1787.v8b_8cd49a_f8f1 github-checks:554.vb_ee03a_000f65 github-oauth:597.ve0c3480fcb_d0 google-oauth-plugin:1.330.vf5e86021cb_ec gson-api:2.10.1-15.v0d99f670e0a_7 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 htmlpublisher:1.33 http_request:1.18 instance-identity:185.v303dc7c645f9 ionicons-api:70.v2959a_b_74e3cf jackson2-api:2.17.0-379.v02de8ec9f64c jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javadoc:243.vb_b_503b_b_45537 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.12 jersey2-api:2.42-147.va_28a_44603b_d5 jjwt-api:0.11.5-112.ve82dfb_224b_a_d jnr-posix-api:3.1.19-2 jobgenerator:1.22 joda-time-api:2.12.7-29.v5a_b_e3a_82269a_ jquery3-api:3.7.1-2 jsch:0.2.16-86.v42e010d9484b_ json-api:20240303-41.v94e11e6de726 json-path-api:2.9.0-58.v62e3e85b_a_655 junit:1265.v65b_14fa_f12f0 kubernetes:4203.v1dd44f5b_1cf9 kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2 kubernetes-credentials:0.11 lockable-resources:1245.vb_05f8a_4e28db_ mailer:470.vc91f60c5d8e2 matrix-auth:3.2.2 matrix-project:822.824.v14451b_c0fd42 metrics:4.2.21-449.v6960d7c54c69 mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd mttr:1.1 oauth-credentials:0.646.v02b_66dc03d2e okhttp-api:4.11.0-172.vda_da_1feeb_c6e pam-auth:1.10 parameterized-scheduler:262.v00f3d90585cc parameterized-trigger:787.v665fcf2a_830b_ permissive-script-security:0.7 pipeline-aws:1.45 pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-github:2.8-159.09e4403bc62f pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:216.vfd8b_ece330ca_ pipeline-graph-view:215.vc0ea_557e4d6f pipeline-groovy-lib:704.vc58b_8890a_384 pipeline-input-step:477.v339683a_8d55e pipeline-milestone-step:119.vdfdc43fc3b_9a_ pipeline-model-api:2.2198.v41dd8ef6dd56 pipeline-model-definition:2.2198.v41dd8ef6dd56 pipeline-model-extensions:2.2198.v41dd8ef6dd56 pipeline-rest-api:2.34 pipeline-stage-step:312.v8cd10304c27a_ pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56 pipeline-stage-view:2.34 pipeline-utility-steps:2.16.2 plain-credentials:179.vc5cb_98f6db_38 plugin-util-api:4.1.0 prism-api:1.29.0-13 project-stats-plugin:23.v47fee1f77b_84 pubsub-light:1.18 run-condition:1.7 saml:4.464.vea_cb_75d7f5e0 scm-api:689.v237b_6d3a_ef7f script-security:1335.vf07d9ce377a_e slack:684.v833089650554 snakeyaml-api:2.2-111.vc6598e30cc65 solarized-theme:0.1 sonar:2.17.2 sse-gateway:1.26 ssh-credentials:337.v395d2403ccd4 sshd:3.322.v159e91f6a_550 structs:337.v1b_04ea_4df7c8 templating-engine:2.5.3 theme-manager:215.vc1ff18d67920 timestamper:1.26 token-macro:400.v35420b_922dcb_ trilead-api:2.142.v748523a_76693 variant:60.v7290fc0eb_b_cd warnings-ng:11.2.2 workflow-api:1291.v51fd2a_625da_7 workflow-basic-steps:1049.v257a_e6b_30fb_d workflow-cps:3894.vd0f0248b_a_fc4 workflow-durable-task-step:1331.vc8c2fed35334 workflow-job:1385.vb_58b_86ea_fff1 workflow-multibranch:773.vc4fe1378f1d5 workflow-scm-step:427.v4ca_6512e7df1 workflow-step-api:657.v03b_e8115821b_ workflow-support:896.v175a_a_9c5b_78f ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Everything runs in kubernetes. Controller is running the Jenkins docker image and agents run in docker:26.1.1-dind docker image

Reproduction steps

It's hard to determine reproduction steps, since the issue happens randomly (but always on the same pipeline)

We have a Jenkinsfile which determines which locks to request.

extraLocks = []
...
extraLocks.push([resource: "alp-${params.branch}-pr-deployment"])
pipeline {
    options {
        lock(resource: "e2e-${params.branch}-${params.domain}", extra: extraLocks)
    }
}
...

Expected Results

I would expect the plugin to be able to release locks properly at the end of the pipeline and not throw an error.

Actual Results

Start of pipeline:

09:05:27  [Pipeline] lock
09:05:27  Trying to acquire lock on [{e2e-SG1-502-circleci-setup-synapse},{alp-SG1-502-circleci-setup-pr-deployment}]
09:05:27  Resource [e2e-SG1-502-circleci-setup-synapse] did not exist. Created.
09:05:27  [{e2e-SG1-502-circleci-setup-synapse},{alp-SG1-502-circleci-setup-pr-deployment}] is not free, waiting for execution ...
09:09:36  Lock acquired on [{e2e-SG1-502-circleci-setup-synapse},{alp-SG1-502-circleci-setup-pr-deployment}]

End of pipeline:

09:12:51  [Pipeline] End of Pipeline
09:12:51  Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 517bb887-4df9-4e73-a3f3-dd873f5c86b3
09:12:51  java.lang.IndexOutOfBoundsException: Index 2 out of bounds for length 2
09:12:51    at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
09:12:51    at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
09:12:51    at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
09:12:51    at java.base/java.util.Objects.checkIndex(Objects.java:361)
09:12:51    at java.base/java.util.ArrayList.get(ArrayList.java:427)
09:12:51    at org.jenkins.plugins.lockableresources.LockableResourcesManager.getNextQueuedContext(LockableResourcesManager.java:778)
09:12:51    at org.jenkins.plugins.lockableresources.LockableResourcesManager.proceedNextContext(LockableResourcesManager.java:684)
09:12:51    at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:662)
09:12:51    at org.jenkins.plugins.lockableresources.LockStepExecution$Callback.finished(LockStepExecution.java:224)
09:12:51    at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:118)
09:12:51    at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:375)
09:12:51    at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:70)
09:12:51    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:144)
09:12:51    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:17)
09:12:51    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:49)
09:12:51    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:180)
09:12:51    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:423)
09:12:51    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:331)
09:12:51    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:295)
09:12:51    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$4(CpsVmExecutorService.java:136)
09:12:51    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
09:12:51    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
09:12:51    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
09:12:51    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
09:12:51    at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
09:12:51    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
09:12:51    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
09:12:51    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
09:12:51    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
09:12:51    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53)
09:12:51    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50)
09:12:51    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
09:12:51    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
09:12:51    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50)
09:12:51    at java.base/java.lang.Thread.run(Thread.java:840)

Anything else?

No response

Are you interested in contributing a fix?

Sure, if I can get a pointer where to start.

mPokornyETM commented 2 months ago

May you pls install newer version.

I provide few fixes in the queue things in the release https://github.com/jenkinsci/lockable-resources-plugin/tree/1246.v28b_e4cc6fa_16

Hopefully it is fixed just now. When not, this "old" stack is helpfull, because the source code looks a "little" bit else now