search

jenkinsci / lockable-resources-plugin

Lock resources against concurrent use
https://plugins.jenkins.io/lockable-resources
MIT License
87 stars 183 forks source link

ConcurrentModificationException when serializing LockedResourcesBuildAction#lockedResources #597

Closed dsilakov closed 8 months ago

dsilakov commented 8 months ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.436 OS: Linux - 4.15.0-121-generic Java: 11.0.21 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- Parameterized-Remote-Trigger:3.2.0 active-directory:2.34 allure-jenkins:3.29.0 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 apache-httpcomponents-client-5-api:5.2.3-1.0 authentication-tokens:1.53.v1c90fd9191a_b_ blueocean:1.27.9 blueocean-bitbucket-pipeline:1.27.9 blueocean-commons:1.27.9 blueocean-config:1.27.9 blueocean-core-js:1.27.9 blueocean-dashboard:1.27.9 blueocean-display-url:2.4.2 blueocean-events:1.27.9 blueocean-git-pipeline:1.27.9 blueocean-github-pipeline:1.27.9 blueocean-i18n:1.27.9 blueocean-jwt:1.27.9 blueocean-personalization:1.27.9 blueocean-pipeline-api-impl:1.27.9 blueocean-pipeline-editor:1.27.9 blueocean-pipeline-scm-api:1.27.9 blueocean-rest:1.27.9 blueocean-rest-impl:1.27.9 blueocean-web:1.27.9 bootstrap5-api:5.3.2-3 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1135.v8de8e7899051 build-name-setter:2.4.0 build-user-vars-plugin:1.9 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.0.2 cloud-stats:320.v96b_65297a_4b_b_ cloudbees-bitbucket-branch-source:856.v04c46c86f911 cloudbees-folder:6.858.v898218f3609d command-launcher:107.v773860566e2e commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-95.v22a_d30ee5d36 config-file-provider:959.vcff671a_4518b_ configuration-as-code:1763.vb_fe9c1b_83f7b credentials:1311.vcf0a_900b_37c2 credentials-binding:642.v737c34dea_6c2 data-tables-api:1.13.8-2 display-url-api:2.200.vb_9327d658781 docker-commons:439.va_3cb_0a_6a_fb_29 docker-java-api:3.3.4-86.v39b_a_5ede342c docker-plugin:1.5 durable-task:523.va_a_22cf15d5e0 echarts-api:5.4.3-2 email-ext:2.102 favorite:2.208.v91d65b_7792a_c folder-properties:1.2.1 font-awesome-api:6.5.1-1 git:5.2.1 git-client:4.6.0 git-parameter:0.9.19 github:1.37.3.1 github-api:1.318-461.v7a_c09c9fa_d63 github-branch-source:1767.va_7d01ea_c7256 gitlab-api:5.3.0-91.v1f9a_fda_d654f gitlab-branch-source:684.vea_fa_7c1e2fe3 gitlab-logo:1.1.2 gitlab-plugin:1.7.16 gson-api:2.10.1-3.vb_25b_599b_e4f8 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 htmlpublisher:1.32 http_request:1.18 instance-identity:185.v303dc7c645f9 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.3-372.v309620682326 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.9 jersey2-api:2.41-133.va_03323b_a_1396 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.87 joda-time-api:2.12.5-5.v5495a_235fedf jquery3-api:3.7.1-1 jsch:0.2.8-65.v052c39de79b_2 json-api:20231013-3.v20f3c247f2fe json-path-api:2.8.0-5.v07cb_a_1ca_738c junit:1252.vfc2e5efa_294f kubernetes:4151.v6fa_f0fb_0b_4c9 kubernetes-client-api:6.8.1-224.vd388fca_4db_3b_ kubernetes-credentials:0.11 lockable-resources:1218.va_3dd45e2b_fa_7 mailer:463.vedf8358e006b_ matrix-auth:3.2.1 matrix-project:818.v7eb_e657db_924 metrics:4.2.18-442.v02e107157925 mina-sshd-api-common:2.11.0-86.v836f585d47fa_ mina-sshd-api-core:2.11.0-86.v836f585d47fa_ multibranch-scan-webhook-trigger:1.0.11 okhttp-api:4.11.0-157.v6852a_a_fa_ec11 ownership:0.13.0 pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:689.veec561a_dee13 pipeline-input-step:477.v339683a_8d55e pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2151.ve32c9d209a_3f pipeline-model-definition:2.2151.ve32c9d209a_3f pipeline-model-extensions:2.2151.ve32c9d209a_3f pipeline-rest-api:2.34 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2151.ve32c9d209a_3f pipeline-stage-view:2.34 pipeline-utility-steps:2.16.0 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.6.0 prism-api:1.29.0-10 prometheus:2.4.1 pubsub-light:1.18 pyenv-pipeline:2.1.2 rebuild:330.v645b_7df10e2a_ role-strategy:689.v731678c3e0eb_ scm-api:683.vb_16722fb_b_80b_ script-security:1294.v99333c047434 snakeyaml-api:2.2-111.vc6598e30cc65 sse-gateway:1.26 ssh-credentials:308.ve4497b_ccd8f4 ssh-slaves:2.945.v90744dcb_47cd sshd:3.312.v1c601b_c83b_0e structs:325.vcb_307d2a_2782 support-core:1366.v9d076592655d timestamper:1.26 token-macro:400.v35420b_922dcb_ trilead-api:2.133.vfb_8a_7b_9c5dd1 uno-choice:2.8.1 variant:60.v7290fc0eb_b_cd workflow-api:1283.v99c10937efcb_ workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3826.v3b_5707fe44da_ workflow-durable-task-step:1313.vcb_970b_d2a_fb_3 workflow-job:1385.vb_58b_86ea_fff1 workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:865.v43e78cc44e0d ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Debian 12

Reproduction steps

Run a pipeline consuming several dozens of lockable resources

Expected Results

Pipeline works fine

Actual Results

From time to time, pipeline fails and we see the following in the end of console log:

java.util.ConcurrentModificationException
        at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
        at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997)
        at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
        at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283)
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270)
Caused: java.lang.RuntimeException: Failed to serialize org.jenkins.plugins.lockableresources.actions.LockedResourcesBuildAction#lockedResources for class org.jenkins.plugins.lockableresources.actions.LockedResourcesBuildAction
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:87)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeBareItem(AbstractCollectionConverter.java:94)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:66)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeCompleteItem(AbstractCollectionConverter.java:81)
        at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:75)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
        at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283)
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270)
Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: a03c949a-e2a7-4da8-ab30-ef46b97e852e
Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
        at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
        at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1303)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1292)
        at com.thoughtworks.xstream.XStream.toXML(XStream.java:1265)
        at hudson.util.XStream2.toXMLUTF8(XStream2.java:386)
        at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:34)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1251)
        at hudson.BulkChange.commit(BulkChange.java:98)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1569)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:512)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:41)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
        at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
[lockable-resources] released lock on [ <a long list of agent names> ]
Finished: FAILURE

Anything else?

This happens relatively rare, but the more lockable resources and labels we have - the higher chance is to get this issues. On Jenkins instance with 500 resources and 80 labels we meet this more frequently then on the instance with 250 resources and 10 labels (though maybe the first instance just runs more pipelines during the same time).

We didn't see this problem with 1141.v7c5f8f31d2ee and jenkins 4.201.2.

Are you interested in contributing a fix?

No response

mPokornyETM commented 8 months ago

Duplicate #503

mPokornyETM commented 8 months ago

@dsilakov to make progress here, it will be very helpfull to review this changes #601 It still does not fixed all the possible races, but it will be big step in the right way. Thx

dsilakov commented 8 months ago

Ok, will try to check, but the problem is that we don't have a stable reproduction and only observed this issue in production environment from time to time.

mPokornyETM commented 8 months ago

@dsilakov Currently it is merged and released, so An plugin update shall be enough

mPokornyETM commented 8 months ago

@dsilakov I will not close this issue, because it still does not fixed all possible exceptions. Next PRs follows soon.

dsilakov commented 5 days ago

Let me reopen this. We have updated lockable-resources to 1246.v28b_e4cc6fa_16 to utilize the new priorities feature and get the same error with the same trace:

all_2024-09-10_09.30.58.log-Also:   java.util.ConcurrentModificationException
all_2024-09-10_09.30.58.log-            at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
all_2024-09-10_09.30.58.log-            at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270)
all_2024-09-10_09.30.58.log:    Caused: java.lang.RuntimeException: Failed to serialize org.jenkins.plugins.lockableresources.actions.LockedResourcesBuildAction#lockedResources for class org.jenkins.plugins.lockableresources.actions.LockedResourcesBuildAction
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:87)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeBareItem(AbstractCollectionConverter.java:94)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:66)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeCompleteItem(AbstractCollectionConverter.java:81)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:75)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270)
all_2024-09-10_09.30.58.log-    Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 2bb0e54b-5e0a-4b52-a3d5-341f9a16bcbe
all_2024-09-10_09.30.58.log-    Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)

The issue appears not often (~1 from 1000 pipelines), but still annoying.

We used 1246.v28b_e4cc6fa_16 since we were able to update to it without touching any other plugins, so plugin version list from the description of this bug is still valid.

Do you think that it makes sense to update some other plugins, as well? Or the reason still lies somewhere in lockable-resources itself?

dsilakov commented 5 days ago

Filed https://github.com/jenkinsci/lockable-resources-plugin/issues/703