Closed aravindavvaru closed 2 weeks ago
Turiok
commented
2 weeks ago Hi @aravindavvaru ,
Can you put the detail of the webhook in Gitlab and the Jenkins response? If possible, add log in Jenkins with "io.jenkins.plugins.gitlabbranchsource" in Debug during the webhook. and add the log in the issue.
aravindavvaru
commented
2 weeks ago Hi @Turiok
Thanks for the response. We were able to find the root cause of the issue and managed to fix the issue. We took a heap dump of the entire Jenkins. We saw that the gitlab trigger threads were in hanging state. Due to our network bandwidth we were able to spawn up to maximum 10 SCM threads. These threads were flooded due to organisational folder job type which were scanning entire gitlab group projects.
--
Jenkins and plugins versions report
Environment
```text Jenkins: 2.440.2 OS: Linux - 5.15.0-1039-aws Java: 21.0.2 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- Office-365-Connector:4.20.0 allure-jenkins-plugin:2.31.1 analysis-model-api:12.3.3 ansicolor:1.0.4 ant:497.v94e7d9fffa_b_9 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 apache-httpcomponents-client-5-api:5.3.1-1.0 artifactory:3.18.4 asm-api:9.7-33.v4d23ef79fcc8 audit-trail:355.v47b_7f827e255 authentication-tokens:1.53.v1c90fd9191a_b_ aws-credentials:218.v1b_e9466ec5da_ aws-java-sdk:1.12.529-406.vdeff15e5817d aws-java-sdk-cloudformation:1.12.529-406.vdeff15e5817d aws-java-sdk-codebuild:1.12.529-406.vdeff15e5817d aws-java-sdk-ec2:1.12.529-406.vdeff15e5817d aws-java-sdk-ecr:1.12.529-406.vdeff15e5817d aws-java-sdk-ecs:1.12.529-406.vdeff15e5817d aws-java-sdk-efs:1.12.529-406.vdeff15e5817d aws-java-sdk-elasticbeanstalk:1.12.529-406.vdeff15e5817d aws-java-sdk-iam:1.12.529-406.vdeff15e5817d aws-java-sdk-kinesis:1.12.529-406.vdeff15e5817d aws-java-sdk-logs:1.12.529-406.vdeff15e5817d aws-java-sdk-minimal:1.12.529-406.vdeff15e5817d aws-java-sdk-secretsmanager:1.12.529-406.vdeff15e5817d aws-java-sdk-sns:1.12.529-406.vdeff15e5817d aws-java-sdk-sqs:1.12.529-406.vdeff15e5817d aws-java-sdk-ssm:1.12.529-406.vdeff15e5817d badge:1.9.1 basic-branch-build-strategies:81.v05e333931c7d blueocean:1.27.5.1 blueocean-bitbucket-pipeline:1.27.5.1 blueocean-commons:1.27.5.1 blueocean-config:1.27.5.1 blueocean-core-js:1.27.5.1 blueocean-dashboard:1.27.5.1 blueocean-display-url:2.4.2 blueocean-events:1.27.5.1 blueocean-git-pipeline:1.27.5.1 blueocean-github-pipeline:1.27.5.1 blueocean-i18n:1.27.5.1 blueocean-jwt:1.27.5.1 blueocean-personalization:1.27.5.1 blueocean-pipeline-api-impl:1.27.5.1 blueocean-pipeline-editor:1.27.5.1 blueocean-pipeline-scm-api:1.27.5.1 blueocean-rest:1.27.5.1 blueocean-rest-impl:1.27.5.1 blueocean-web:1.27.5.1 bootstrap5-api:5.3.2-2 bouncycastle-api:2.29 branch-api:2.1163.va_f1064e4a_a_f3 build-monitor-plugin:1.14-744.v35fd6fa_a_26b_2 build-name-setter:2.4.0 build-timeout:1.31 build-with-parameters:76.v9382db_f78962 built-on-column:1.4 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.0.2 cloudbees-bitbucket-branch-source:848.v42c6a_317eda_e cloudbees-disk-usage-simple:187.v6378d330d1d4 cloudbees-folder:6.858.v898218f3609d cobertura:1.17 code-coverage-api:4.10.0 command-launcher:107.v773860566e2e commons-httpclient3-api:3.1-3 commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-95.v22a_d30ee5d36 config-file-provider:959.vcff671a_4518b_ configuration-as-code:1775.v810dc950b_514 copyartifact:722.v0662a_9b_e22a_c credentials:1337.v60b_d7b_c7b_c9f credentials-binding:642.v737c34dea_6c2 crowd2:4.1.0 dashboard-view:2.495.v07e81500c3f2 data-tables-api:1.13.6-5 display-url-api:2.200.vb_9327d658781 docker-commons:439.va_3cb_0a_6a_fb_29 docker-workflow:572.v950f58993843 downstream-build-cache:1.7 dtkit-api:3.0.2 durable-task:550.v0930093c4b_a_6 ec2:2.0.7 ec2-fleet:3.0.0 echarts-api:5.4.0-7 email-ext:2.102 emailext-template:1.5 embeddable-build-status:412.v09da_db_1dee68 favorite:2.4.3 font-awesome-api:6.5.1-2 forensics-api:2.3.0 generic-webhook-trigger:1.88.2 git:5.2.1 git-client:4.5.0 git-parameter:0.9.19 github:1.37.3 github-api:1.316-451.v15738eef3414 github-branch-source:1741.va_3028eb_9fd21 gitlab-api:5.3.0-91.v1f9a_fda_d654f gitlab-branch-source:703.vc5fd7effcc6b_ gitlab-plugin:1.7.14 gradle:2.8.2 groovy:457.v99900cb_85593 groovy-postbuild:2.5 gson-api:2.10.1-15.v0d99f670e0a_7 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 htmlpublisher:1.32 influxdb:3.5 instance-identity:173.va_37c494ec4e5 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.3-363.v82c51b_de9f60 jacoco:3.3.4 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:243.vb_b_503b_b_45537 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.5.1 jersey2-api:2.40-1 jira:3.12 jira-steps:2.0.165.v8846cf59f3db jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.18-1 job-dsl:1.84 job-restrictions:0.8 jobConfigHistory:1229.v3039470161a_d joda-time-api:2.12.7-29.v5a_b_e3a_82269a_ jquery3-api:3.7.1-1 jsch:0.2.8-65.v052c39de79b_2 json-api:20240303-41.v94e11e6de726 json-path-api:2.9.0-58.v62e3e85b_a_655 junit:1240.vf9529b_881428 locale:314.v22ce953dfe9e lockable-resources:1185.v0c528656ce04 mailer:463.vedf8358e006b_ material-theme:0.5.2-rc100.6121925fe229 matrix-auth:3.2.1 matrix-project:818.v7eb_e657db_924 mattermost:3.1.3 maven-plugin:3.23 mercurial:1260.vdfb_723cdcc81 metrics:4.2.18-442.v02e107157925 mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ monitoring:1.95.0 next-build-number:1.8 next-executions:179.vb_e011d7e3b_64 node-iterator-api:55.v3b_77d4032326 oic-auth:2.6 okhttp-api:4.11.0-157.v6852a_a_fa_ec11 parameterized-scheduler:255.v73827fcdf618 parameterized-trigger:2.46 pipeline-build-step:505.v5f0844d8d126 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-graph-view:202.v6da_a_9e590325 pipeline-groovy-lib:689.veec561a_dee13 pipeline-input-step:477.v339683a_8d55e pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2144.v077a_d1928a_40 pipeline-model-definition:2.2144.v077a_d1928a_40 pipeline-model-extensions:2.2144.v077a_d1928a_40 pipeline-rest-api:2.33 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40 pipeline-stage-view:2.33 pipeline-utility-steps:2.16.0 plain-credentials:143.v1b_df8b_d3b_e48 plugin-usage-plugin:4.2 plugin-util-api:4.1.0 popper2-api:2.11.6-2 prism-api:1.29.0-8 prometheus:2.3.3 pubsub-light:1.17 rebuild:320.v5a_0933a_e7d61 resource-disposer:0.23 robot:3.4.0 role-strategy:689.v731678c3e0eb_ scm-api:690.vfc8b_54395023 scm-filter-branch-pr:77.vf2b_85e1970d5 script-security:1326.vdb_c154de8669 snakeyaml-api:2.2-111.vc6598e30cc65 sse-gateway:1.26 ssh-agent:333.v878b_53c89511 ssh-credentials:308.ve4497b_ccd8f4 ssh-slaves:2.916.vd17b_43357ce4 sshd:3.312.v1c601b_c83b_0e structs:337.v1b_04ea_4df7c8 support-core:1371.v709cef748b_d0 synopsys-coverity:3.0.3 theme-manager:211.vef2a_42c645a_b_ throttle-concurrents:2.14 timestamper:1.26 token-macro:384.vf35b_f26814ec trilead-api:2.84.v72119de229b_7 uno-choice:2.8.0 variant:60.v7290fc0eb_b_cd warnings-ng:10.5.2 workflow-aggregator:596.v8c21c963d92d workflow-api:1291.v51fd2a_625da_7 workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3853.vb_a_490d892963 workflow-durable-task-step:1289.v4d3e7b_01546b_ workflow-job:1385.vb_58b_86ea_fff1 workflow-multibranch:773.vc4fe1378f1d5 workflow-scm-step:415.v434365564324 workflow-step-api:657.v03b_e8115821b_ workflow-support:865.v43e78cc44e0d ws-cleanup:0.45 xunit:3.1.3 yet-another-build-visualizer:1.16 ```What Operating System are you using (both controller, and any agents involved in the problem)?
Distributor ID: Ubuntu Description: Ubuntu 20.04.6 LTS Release: 20.04 Codename: focal
Reproduction steps
step 1: create a multibranch pipeline with gitlab-branch source step 2: The jenkins webhook is registered on gitlab.
Step 3: Create a merge request in the repo configured in step 1 Step 4: The merge request event is received by the jenkins but nothing happens.
Expected Results
Merge request event triggers the build immediately.
Actual Results
Jenkins instance do not trigger the build, intermittently it triggers the build
Anything else?
It happens on instances having more than 8 multibranch pipelines.
Are you interested in contributing a fix?
No response