search

jenkinsci / violation-comments-to-stash-plugin

Comments Bitbucket Server (or Stash) pull requests with static code analyzer findings.
https://plugins.jenkins.io/violation-comments-to-stash
MIT License
43 stars 16 forks source link

Jenkins agents becoming unresponsive during violations run #76

Open AndLindemann opened 2 years ago

AndLindemann commented 2 years ago

Jenkins and plugins versions report

Jenkins: 2.319.2 OS: Linux - 5.11.0-1025-aws

ace-editor:1.1 active-directory:2.25.1 analysis-model-api:10.9.1 ant:1.13 antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 authentication-tokens:1.4 authorize-project:1.4.0 aws-credentials:1.33 aws-java-sdk-ec2:1.12.131-302.vbef9650c6521 aws-java-sdk-minimal:1.12.131-302.vbef9650c6521 bitbucket:214.v2fd4234d0554 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-4 bouncycastle-api:2.25 branch-api:2.7.0 build-name-setter:2.2.0 build-timeout:1.20 built-on-column:1.1 caffeine-api:2.9.2-29.v717aac953ff3 checks-api:1.7.2 cloudbees-bitbucket-branch-source:751.vda_24678a_f781 cloudbees-folder:6.17 cobertura:1.17 code-coverage-api:2.0.4 command-launcher:1.6 conditional-buildstep:1.4.1 config-file-provider:3.8.2 configuration-as-code:1.55.1 copyartifact:1.46.2 credentials:1074.v60e6c29b_b44b credentials-binding:1.27.1 dashboard-view:2.18 data-tables-api:1.11.3-6 display-url-api:2.3.5 dtkit-api:3.0.0 durable-task:493.v195aefbb0ff2 ec2:1.66 echarts-api:5.2.2-2 email-ext:2.86 emailext-template:1.4 embeddable-build-status:2.0.3 envinject:2.4.0 envinject-api:1.8 external-monitor-job:1.7 font-awesome-api:5.15.4-5 forensics-api:1.7.0 git:4.10.2 git-client:3.11.0 git-forensics:1.3.0 git-parameter:0.9.15 git-server:1.10 github:1.34.1 github-api:1.301-378.v9807bd746da5 github-branch-source:2.11.4 gradle:1.38 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-1.0 htmlpublisher:1.28 jackson2-api:2.13.1-246.va8a9f3eaf46a javadoc:1.6 javax-activation-api:1.2.0-2 javax-mail-api:1.6.2-5 jaxb:2.3.0.1 jdk-tool:1.5 jenkins-multijob-plugin:1.36 jjwt-api:0.11.2-9.c8b45b8bb173 jquery:1.12.4-1 jquery3-api:3.6.0-2 jsch:0.1.55.2 junit:1.53 ldap:2.7 locale:1.4 lockable-resources:2.13 mailer:408.vd726a_1130320 matrix-auth:3.0 matrix-project:1.20 maven-plugin:3.16 mercurial:2.16 momentjs:1.1.1 node-iterator-api:1.5.1 nodejs:1.5.1 okhttp-api:4.9.3-105.vb96869f8ac3a pam-auth:1.6.1 parameterized-trigger:2.43 pipeline-build-step:2.15 pipeline-github-lib:1.0 pipeline-graph-analysis:188.v3a01e7973f2c pipeline-input-step:427.va6441fa17010 pipeline-milestone-step:1.3.2 pipeline-model-api:1.9.3 pipeline-model-definition:1.9.3 pipeline-model-extensions:1.9.3 pipeline-rest-api:2.20 pipeline-stage-step:291.vf0a8a7aeeb50 pipeline-stage-tags-metadata:1.9.3 pipeline-stage-view:2.20 plain-credentials:1.7 plugin-util-api:2.12.0 popper-api:1.16.1-2 popper2-api:2.11.2-1 prism-api:1.25.0-2 publish-over:0.22 rebuild:1.33 resource-disposer:0.17 role-strategy:3.2.0 run-condition:1.5 scm-api:595.vd5a_df5eb_0e39 script-security:1131.v8b_b_5eda_c328e snakeyaml-api:1.29.1 ssh:2.6.1 ssh-agent:1.24.1 ssh-credentials:1.19 ssh-slaves:1.33.0 sshd:3.1.0 structs:308.v852b473a2b8c throttle-concurrents:2.6 timestamper:1.16 token-macro:267.vcdaea6462991 trilead-api:1.0.13 variant:1.4 violation-comments-to-stash:1.128 warnings-ng:9.11.0 windows-slaves:1.8 workflow-aggregator:2.6 workflow-api:1122.v7a_916f363c86 workflow-basic-steps:2.24 workflow-cps:2648.va9433432b33c workflow-cps-global-lib:552.vd9cc05b8a2e1 workflow-durable-task-step:1121.va_65b_d2701486 workflow-job:1145.v7f2433caa07f workflow-multibranch:706.vd43c65dec013 workflow-scm-step:2.13 workflow-step-api:622.vb_8e7c15bc95a workflow-support:813.vb_d7c3d2984a_0 ws-cleanup:0.40 xunit:3.0.5

What Operating System are you using (both controller, and any agents involved in the problem)?

Ubuntu Focal 20.04 LTS Server on Jenkins Controller Debian Bullseye on Agents

Reproduction steps

Running violations to bitbucket in the pipeline very often causes agents to go unresponsive, missing pings and getting disconnected from the master.

                ViolationsToBitbucketServer([
                    bitbucketServerUrl: 'https://ourbitbucketserver/',
                    commentOnlyChangedContent: true,
                    commentOnlyChangedContentContext: 5,
                    commentOnlyChangedFiles: true,
                    createCommentWithAllSingleFileComments: false,
                    createSingleFileComments: true,
                    maxNumberOfViolations: 100,
                    keepOldComments: false,
                    projectKey: 'theKey',
                    pullRequestId: env.CHANGE_ID,
                    repoSlug: 'repo',
                    credentialsId: 'cred-id',

                    commentTemplate: """
                    {{#violation.rule}}Rule: {{violation.rule}}{{/violation.rule}}
                    {{violation.message}}
                    """.stripIndent(),

                    violationConfigs: [
                        [parser: 'CHECKSTYLE', pattern: '.*checkstyle\\.xml\$', reporter: 'Checkstyle']
                    ]
                ])

Expected Results

Plugin not getting stuck somewhere

Actual Results

Plugin gets stuck without any indication in logs as to why this is happening, no errors in log. It often seems to happen after the "Files with violations" list (the list of these files is often the last output in the logs).

Anything else?

Any way to further debug this or enable a verbose mode to see at which steps the plugin gets stuck for what reasons?

tomasbjerre commented 2 years ago

Perhaps try reducing this number:

maxNumberOfViolations: 100,

Perhaps 10 is enough.

AndLindemann commented 2 years ago

I have tried reducing maxNumberOfViolations, but that doesn't seem to help. I'm generally seeing it more often if a large number of violations is in the checkstyle reports. Due to the sheer size of our project, it is not uncommon to have like 400+ violations in the phpstan/checkstyle reports.

Most of those violations are usually outside of the actual pull request changes so would not end up as comments, but as we run phpcs/phpstan over the whole project, it still generates the reports for everything of course and the plugin then would have to sort out what is relevant to the PR and what is not.

Is this too much for the plugin to handle?

tomasbjerre commented 2 years ago

You may also try using the command line tool. It hase similar naming of configuration options.

https://github.com/tomasbjerre/violation-comments-to-bitbucket-server-command-line