search

jenkinsci / lockable-resources-plugin

Lock resources against concurrent use
https://plugins.jenkins.io/lockable-resources
MIT License
87 stars 185 forks source link

ConcurrentModificationException when serializing LockedResourcesBuildAction#lockedResources #703

Closed dsilakov closed 2 months ago

dsilakov commented 2 months ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.436 OS: Linux - 4.15.0-121-generic Java: 11.0.21 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- Parameterized-Remote-Trigger:3.2.0 active-directory:2.34 allure-jenkins:3.29.0 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 apache-httpcomponents-client-5-api:5.2.3-1.0 authentication-tokens:1.53.v1c90fd9191a_b_ blueocean:1.27.9 blueocean-bitbucket-pipeline:1.27.9 blueocean-commons:1.27.9 blueocean-config:1.27.9 blueocean-core-js:1.27.9 blueocean-dashboard:1.27.9 blueocean-display-url:2.4.2 blueocean-events:1.27.9 blueocean-git-pipeline:1.27.9 blueocean-github-pipeline:1.27.9 blueocean-i18n:1.27.9 blueocean-jwt:1.27.9 blueocean-personalization:1.27.9 blueocean-pipeline-api-impl:1.27.9 blueocean-pipeline-editor:1.27.9 blueocean-pipeline-scm-api:1.27.9 blueocean-rest:1.27.9 blueocean-rest-impl:1.27.9 blueocean-web:1.27.9 bootstrap5-api:5.3.2-3 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1135.v8de8e7899051 build-name-setter:2.4.0 build-user-vars-plugin:1.9 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.0.2 cloud-stats:320.v96b_65297a_4b_b_ cloudbees-bitbucket-branch-source:856.v04c46c86f911 cloudbees-folder:6.858.v898218f3609d command-launcher:107.v773860566e2e commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-95.v22a_d30ee5d36 config-file-provider:959.vcff671a_4518b_ configuration-as-code:1763.vb_fe9c1b_83f7b credentials:1311.vcf0a_900b_37c2 credentials-binding:642.v737c34dea_6c2 data-tables-api:1.13.8-2 display-url-api:2.200.vb_9327d658781 docker-commons:439.va_3cb_0a_6a_fb_29 docker-java-api:3.3.4-86.v39b_a_5ede342c docker-plugin:1.5 durable-task:523.va_a_22cf15d5e0 echarts-api:5.4.3-2 email-ext:2.102 favorite:2.208.v91d65b_7792a_c folder-properties:1.2.1 font-awesome-api:6.5.1-1 git:5.2.1 git-client:4.6.0 git-parameter:0.9.19 github:1.37.3.1 github-api:1.318-461.v7a_c09c9fa_d63 github-branch-source:1767.va_7d01ea_c7256 gitlab-api:5.3.0-91.v1f9a_fda_d654f gitlab-branch-source:684.vea_fa_7c1e2fe3 gitlab-logo:1.1.2 gitlab-plugin:1.7.16 gson-api:2.10.1-3.vb_25b_599b_e4f8 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 htmlpublisher:1.32 http_request:1.18 instance-identity:185.v303dc7c645f9 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.3-372.v309620682326 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.9 jersey2-api:2.41-133.va_03323b_a_1396 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.87 joda-time-api:2.12.5-5.v5495a_235fedf jquery3-api:3.7.1-1 jsch:0.2.8-65.v052c39de79b_2 json-api:20231013-3.v20f3c247f2fe json-path-api:2.8.0-5.v07cb_a_1ca_738c junit:1252.vfc2e5efa_294f kubernetes:4151.v6fa_f0fb_0b_4c9 kubernetes-client-api:6.8.1-224.vd388fca_4db_3b_ kubernetes-credentials:0.11 lockable-resources:1246.v28b_e4cc6fa_16 mailer:463.vedf8358e006b_ matrix-auth:3.2.1 matrix-project:818.v7eb_e657db_924 metrics:4.2.18-442.v02e107157925 mina-sshd-api-common:2.11.0-86.v836f585d47fa_ mina-sshd-api-core:2.11.0-86.v836f585d47fa_ multibranch-scan-webhook-trigger:1.0.11 okhttp-api:4.11.0-157.v6852a_a_fa_ec11 ownership:0.13.0 pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:689.veec561a_dee13 pipeline-input-step:477.v339683a_8d55e pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2151.ve32c9d209a_3f pipeline-model-definition:2.2151.ve32c9d209a_3f pipeline-model-extensions:2.2151.ve32c9d209a_3f pipeline-rest-api:2.34 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2151.ve32c9d209a_3f pipeline-stage-view:2.34 pipeline-utility-steps:2.16.0 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.6.0 prism-api:1.29.0-10 prometheus:2.4.1 pubsub-light:1.18 pyenv-pipeline:2.1.2 rebuild:330.v645b_7df10e2a_ role-strategy:689.v731678c3e0eb_ scm-api:683.vb_16722fb_b_80b_ script-security:1294.v99333c047434 snakeyaml-api:2.2-111.vc6598e30cc65 sse-gateway:1.26 ssh-credentials:308.ve4497b_ccd8f4 ssh-slaves:2.945.v90744dcb_47cd sshd:3.312.v1c601b_c83b_0e structs:325.vcb_307d2a_2782 support-core:1366.v9d076592655d timestamper:1.26 token-macro:400.v35420b_922dcb_ trilead-api:2.133.vfb_8a_7b_9c5dd1 uno-choice:2.8.1 variant:60.v7290fc0eb_b_cd workflow-api:1283.v99c10937efcb_ workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3826.v3b_5707fe44da_ workflow-durable-task-step:1313.vcb_970b_d2a_fb_3 workflow-job:1385.vb_58b_86ea_fff1 workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:865.v43e78cc44e0d ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Debian 12

Reproduction steps

Run a pipeline consuming several dozens of lockable resources.

Several hundreds such pipelines a day often trigger such an issue

Expected Results

Pipeline works fine

Actual Results

From time to time, pipeline fails and we see the following in the end of console log:

all_2024-09-10_09.30.58.log-Also:   java.util.ConcurrentModificationException
all_2024-09-10_09.30.58.log-            at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
all_2024-09-10_09.30.58.log-            at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270)
all_2024-09-10_09.30.58.log:    Caused: java.lang.RuntimeException: Failed to serialize org.jenkins.plugins.lockableresources.actions.LockedResourcesBuildAction#lockedResources for class org.jenkins.plugins.lockableresources.actions.LockedResourcesBuildAction
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:87)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeBareItem(AbstractCollectionConverter.java:94)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:66)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeCompleteItem(AbstractCollectionConverter.java:81)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:75)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270)
all_2024-09-10_09.30.58.log-    Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 2bb0e54b-5e0a-4b52-a3d5-341f9a16bcbe
all_2024-09-10_09.30.58.log-    Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226)
all_2024-09-10_09.30.58.log-            at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
all_2024-09-10_09.30.58.log-            at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)

Unfortunately this breaks BlueOcean representation for the pipelines.

Anything else?

https://github.com/jenkinsci/lockable-resources-plugin/issues/597 was about the same problem but with older plugin version.

We updated to 1246.v28b_e4cc6fa_16 since we were able to update to it without touching any other plugins, so plugin version list from the description of this bug is still valid. This version should have all fixes mentioned in https://github.com/jenkinsci/lockable-resources-plugin/issues/597

Do you think that it makes sense to update some other plugins, as well? Or the reason still lies somewhere in lockable-resources itself?

Are you interested in contributing a fix?

No response

mPokornyETM commented 2 months ago

Oh no, pls not again. :-( But good news, I found the same exception by fixing this PR #673

We use the changes over 3 mounts and does not find some side effects. When you agree, I can merge the PR (without review again :-( ) and you need just update the plugin.

Back to your question. This is failure in this plugin, and you does not need to update some other plugins.

It will be great, when you install this version on your Jenkins instance https://ci.jenkins.io/job/Plugins/job/lockable-resources-plugin/job/PR-673/11/artifact/org/6wind/jenkins/lockable-resources/1329.v19c14cb_b_d796/ to verify, that ot works. In that case pls write some hit in the PR, that it is also verified by someone else (for transparency)

Anyway: just give me feadback and I will try to merge it into master, so our issues will be fixed ASAP ;-)

dsilakov commented 2 months ago

Ok, will give it a try, thanks

dsilakov commented 2 months ago

1329.v19c14cb_b_d796 has been installed. Running for 5 hours up to now without serialization issues or other problems, but I think it makes sense to monitor for longer time

mPokornyETM commented 2 months ago

1329.v19c14cb_b_d796 has been installed. Running for 5 hours up to now without serialization issues or other problems, but I think it makes sense to monitor for longer time

Let me know next week. When it works over few days without issues on your side, I will megre it into official master.

Thx for your feadback

dsilakov commented 2 months ago

So far so good, but we started to get a lot of messages about failures to save LR state:

image

Do you have any ideas what can cause this?

mPokornyETM commented 2 months ago

No, We does not save the LRM state. It cost too much time for us. But I will try to check it.

mPokornyETM commented 2 months ago

I analysed the changes now. I does not see reason, why it might fail the save() function. @dsilakov Can you check the persmission om your jenkins instance. Like '''ls -la''' Maybe somebody rewrite the file content and the standard locale Jenkins user has no permission to rewrite it now.

dsilakov commented 2 months ago

Yes, somebody constantly resets ownership of this file. If I change owner:group to jenkins:jenkins, the error goes away, however very soon the permissions are reset to root:jenkins. I'll try to figure out who can it be, but the thing is that we have the same root:jenkins permissions on other jenkins instances with older plugins and this doesn't lead to any problems. One more point is that in order to update LR, we had to update a bunch of other plugins, maybe some of that update influences our scenario. Will check. But as for original problem with serialization - likely yes, the new version solves it, never seen it again.

mPokornyETM commented 2 months ago

Great news, I will merge it now into master.