ravage84
commented
1 year ago I observe an issue that looks like this one. A scan triggered every minute is now running for over a week.
From the log:
Build was timed triggered. {Translated from German}
[Mi. Jan. 18 23:49:06 UTC 2023] Starting branch indexing...
Checking branches..
{branches got detected}
5 branches were processed
Checking merge requests..
0 merge requests were processed
Checking tags..
{some tags got detected}
Checking tag {random tag}
sIuv
grazius
redgkl77
Jenkins and plugins versions report
Environment
``` OS: Windows Server 2019 - 10.0 --- Office-365-Connector:4.15.2 ace-editor:1.1 active-directory:2.25.1 analysis-model-api:10.9.4 ansicolor:1.0.1 ant:1.13 antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 artifact-manager-s3:670.v0558a_cb_c82c2 audit-trail:3.10 authentication-tokens:1.4 authorize-project:1.4.0 aws-credentials:191.vcb_f183ce58b_9 aws-global-configuration:106.v106dc1d8d86e aws-java-sdk-ec2:1.12.287-357.vf82d85a_6eefd aws-java-sdk-minimal:1.12.287-357.vf82d85a_6eefd badge:1.9.1 bitbucket:223.vd12f2bca5430 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-6 bouncycastle-api:2.25 branch-api:2.1044.v2c007e51b_87f build-blocker-plugin:1.7.8 build-discarder:60.v1747b0eb632a build-timeout:1.20 build-timestamp:1.0.3 caffeine-api:2.9.3-65.v6a_47d0f4d1fe cctray-xml:1.1 checks-api:1.7.4 cloudbees-bitbucket-branch-source:757.vddedc5f2589a_ cloudbees-folder:6.714.v79e858ef76a_2 command-launcher:1.6 conditional-buildstep:1.4.2 config-file-provider:3.10.0 configurationslicing:430.v966357576543 credentials:1087.1089.v2f1b_9a_b_040e4 credentials-binding:1.27.1 data-tables-api:1.11.4-4 display-url-api:2.3.6 docker-commons:1.19 docker-workflow:1.28 durable-task:495.v29cd95ec10f2 echarts-api:5.3.3-1 email-ext:2.87 embeddable-build-status:2.0.4 envinject:2.839.v52c702c10635 envinject-api:1.180.v98d833b_27470 extended-read-permission:3.2 external-monitor-job:191.v363d0d1efdf8 font-awesome-api:6.0.0-1 forensics-api:1.12.0 git:4.11.3 git-client:3.11.0 git-server:1.10 github:1.34.3 github-api:1.301-378.v9807bd746da5 github-branch-source:1598.v91207e9f9b_4a_ gitlab-api:5.0.1-72.vb_8c272862e86 gitlab-branch-source:625.v85cf3a_400cfe gitlab-plugin:1.5.35 global-slack-notifier:1.5 gradle:1.38 groovy:2.4 groovy-postbuild:2.5 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 jackson2-api:2.13.3-285.vc03c0256d517 javadoc:217.v905b_86277a_2a_ javax-activation-api:1.2.0-3 javax-mail-api:1.6.2-5 jaxb:2.3.6-1 jdk-tool:1.5 jersey2-api:2.36-2 jjwt-api:0.11.2-9.c8b45b8bb173 jnr-posix-api:3.1.7-3 job-dsl:1.79 jquery:1.12.4-1 jquery-detached:1.2.1 jquery-ui:1.0.2 jquery3-api:3.6.0-4 jsch:0.1.55.2 junit:1119.1121.vc43d0fc45561 ldap:2.8 ldapemail:0.8 lockable-resources:2.14 mailer:408.vd726a_1130320 mapdb-api:1.0.9.0 matrix-auth:2.6.11 matrix-project:771.v574584b_39e60 maven-plugin:3.18 mercurial:2.16.2 momentjs:1.1.1 monitoring:1.90.0 msbuild:1.30 mstest:1.0.0 multibranch-scan-webhook-trigger:1.0.9 next-build-number:1.8 okhttp-api:4.9.3-105.vb96869f8ac3a p4:1.12.2 pam-auth:1.7 parameterized-scheduler:1.0 permissive-script-security:0.7 persistent-parameter:1.3 pipeline-build-step:2.16 pipeline-github-lib:36.v4c01db_ca_ed16 pipeline-graph-analysis:188.v3a01e7973f2c pipeline-groovy-lib:593.va_a_fc25d520e9 pipeline-input-step:449.v77f0e8b_845c4 pipeline-milestone-step:100.v60a_03cd446e1 pipeline-model-api:2.2064.v5eef7d0982b_e pipeline-model-declarative-agent:1.1.1 pipeline-model-definition:2.2064.v5eef7d0982b_e pipeline-model-extensions:2.2064.v5eef7d0982b_e pipeline-rest-api:2.23 pipeline-stage-step:291.vf0a8a7aeeb50 pipeline-stage-tags-metadata:2.2064.v5eef7d0982b_e pipeline-stage-view:2.23 pipeline-utility-steps:2.12.0 plain-credentials:1.8 plugin-util-api:2.17.0 popper-api:1.16.1-2 popper2-api:2.11.4-1 powershell:1.7 prism-api:1.26.0-2 publish-over:0.22 publish-over-cifs:0.16 pvs-studio:7.18 resource-disposer:0.18 role-strategy:484.v8a_a_e4b_d785fd run-condition:1.5 rundeck:3.6.11 schedule-build:244.vce2a_2e926127 scm-api:608.vfa_f971c5a_a_e9 script-security:1175.v4b_d517d6db_f0 show-build-parameters:1.0 simple-theme-plugin:103.va_161d09c38c7 skip-certificate-check:1.0 slack:608.v19e3b_44b_b_9ff snakeyaml-api:1.29.1 ssh-agent:1.24.1 ssh-credentials:277.v95c2fec1c047 ssh-slaves:1.806.v2253cedd3295 sshd:3.1.0 strict-crumb-issuer:2.1.0 structs:324.va_f5d6774f3a_d subversion:2.15.5 throttle-concurrents:2.7 timestamper:1.17 token-macro:293.v283932a_0a_b_49 trilead-api:1.0.13 unity3d-plugin:1.3 variant:1.4 vstestrunner:1.0.8 warnings-ng:9.11.1 windows-slaves:1.8.1 workflow-aggregator:2.7 workflow-api:1200.v8005c684b_a_c6 workflow-basic-steps:941.vdfe1b_a_132c64 workflow-cps:2725.v7b_c717eb_12ce workflow-cps-global-lib:588.v576c103a_ff86 workflow-durable-task-step:1128.v8c259d125340 workflow-job:1174.vdcb_d054cf74a_ workflow-multibranch:711.vdfef37cda_816 workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:625.vd896b_f445a_f8 workflow-support:820.vd1a_6cc65ef33 zulip:1.3.0 ```What Operating System are you using (both controller, and any agents involved in the problem)?
OS: Windows Server 2019 (Controller / Agent Both)
Reproduction steps
Step1: Create Multibranch Pipeline Job Step2: While scanning branch indexing, Gitlab SCM Server occur Step3: Scanning Task Hang
Expected Results
After minitues, scanning task exception or timeout exception
Actual Results
while fetching branch indexing in a multibrach pipeline job, it getting stuck at scanning branch for days (As shown in the attached image)
Problmatic thread got stuck in SocketInputStream.socketread0 method(this thread didn't kill by groovy script interrupt or monitoring plugin thread kill/stop)
Anything else?
Timeouts setting(org.gitlab4j.api.GitLabApiClien setRequestTimeout API call in gitlabbranchsource plugin) are needed to solve the above problem.
The same issue can be identified in Bitbuckt/Github source plugin(JENKINS-36724](https://issues.jenkins.io/browse/JENKINS-36724), Bitbucket Hang Timeout Setting Code Commit)
Additonally, it is necessary to provide a jvm system property to set the gitlab timeout for timeout value.(example, java -Dio.jenkins.plugins.gitlabbranchsource.config.readTimeout -Dio.jenkins.plugins.gitlabbranchsource.config.connectTimeout)
If you can, we propose a common SCMSourceRequest interface. This issue occurs in many jenkins sCM source plugin such as gitblab/bitbucket/github, so we need a common interface at Jenkins level that sets the request timeout based on system properties