timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-43106] pipeline job hangs forever at checkout GitSCM - after git rev-list (sporadic) #3961

Closed timja closed 6 years ago

timja commented 7 years ago

Sometimes a pipeline job hangs around checkout. 

Lately this seems to be after git rev-list. 

 

 > git rev-list --no-walk db25209b84b12cca5214227a7999fe280c82017e # timeout=10

It will hang here until job is killed. A restart is fine - and it only happens occasionally. 

This has been more commonly reported since 2018 February/January, a recent update. 

 

A support bundle of this when it was happening: support_2018-02-07_22.54.25.zip

IT was happening on slave-05 at the time (can see in its thread dump a few waiting things). Timeout does not work, it will wait indefinitely. 

 

 


My pipeline job is getting sporadically hung on checkout step.

Our git repo is relatively small and clone should not take more than 1 min.

The problem occurring really often once a couple builds. Once it's occurring, the only way that I know to get rid of this state so far is, is disconnect the slave and re-connect it back.

Build log:

Started by user
Checking out git to read Jenkinsfile.bootstrap
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url # timeout=10
Fetching upstream changes from
> git --version # timeout=10
using GIT_SSH to set credentials jenkinsslave SSH key
> git fetch --tags --progress +refs/heads/:refs/remotes/origin/
> git rev-parse refs/remotes/origin/devops_jenkins^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/devops_jenkins^{commit} # timeout=10
Checking out Revision 2fd998fbe8b23f814b57d8c6eb7b46004268eedc (refs/remotes/origin/devops_jenkins)
> git config core.sparsecheckout # timeout=10
> git checkout -f 2fd998fbe8b23f814b57d8c6eb7b46004268eedc
> git rev-list 2fd998fbe8b23f814b57d8c6eb7b46004268eedc # timeout=10
[Pipeline] timestamps
[Pipeline] {
[Pipeline] node
00:00:01.249 Running on in
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Load Jenkinsfile)
[Pipeline] echo
00:00:01.267 Branch: origin/devops_jenkins Repo:
[Pipeline] checkout
00:03:27.565 > git rev-parse -is-inside-work-tree # timeout=10 <- This and further lines are being printed only after slave disconnect.
00:03:27.566 Fetching changes from the remote Git repository
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] End of Pipeline
hudson.remoting.RemotingSystemException: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:272)
at com.sun.proxy.$Proxy98.setRemoteUrl(Unknown Source)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:794)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1070)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1101)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:221)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:191)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:256)
... 16 more
Caused by: hudson.remoting.Channel$OrderlyShutdown
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1121)
at hudson.remoting.Channel$1.handle(Channel.java:526)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
Caused by: Command close created at
at hudson.remoting.Command.(Command.java:59)
at hudson.remoting.Channel$CloseCommand.(Channel.java:1115)
at hudson.remoting.Channel$CloseCommand.(Channel.java:1113)
at hudson.remoting.Channel.close(Channel.java:1273)
at hudson.remoting.Channel.close(Channel.java:1255)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1120)
... 2 more
Finished: FAILURE


Originally reported by levt, imported from: pipeline job hangs forever at checkout GitSCM - after git rev-list (sporadic)
  • assignee: vrobert78
  • status: Closed
  • priority: Blocker
  • resolution: Duplicate
  • resolved: 2018-02-12T13:46:15+00:00
  • imported: 2022/01/10
timja commented 7 years ago

levt:

My environment is also affected by https://issues.jenkins-ci.org/browse/JENKINS-43038 

I suspect the both problem have similar root cause that could be related to communication between master and slave.

timja commented 7 years ago

jamesdumay:

levt are you still having issues, even upgrading to a new Jenkins core and updating all plugins?

timja commented 7 years ago

bparker:

This is a problem that I see on a daily basis. It just hangs at the git cloning step. 

We are running Ubuntu 14.04.5, Jenkins 2.70, JRE 1.8.0_131-b11 and git 2.11.0 

We are using the following plugins:

PAM Authentication plugin (pam-auth): 1.3
Branch API Plugin (branch-api): 2.0.11
TextFinder plugin (text-finder): 1.10
Gitlab Merge Request Builder (gitlab-merge-request-jenkins): 2.0.0
Docker Commons Plugin (docker-commons): 1.8
skip-certificate-check (skip-certificate-check): 1.0
JIRA Pipeline Steps (jira-steps): 1.2.3
Docker Pipeline (docker-workflow): 1.12
Subversion Plug-in (subversion): 2.9
Next Build Number Plugin (next-build-number): 1.4
Lockable Resources plugin (lockable-resources): 2.0
Pipeline: Multibranch (workflow-multibranch): 2.16
GitLab Plugin (gitlab-plugin): 1.4.7
Git Parameter Plug-In (git-parameter): 0.8.0
Changes since last successfull build Plugin (changes-since-last-success): 0.5
Amazon EC2 plugin (ec2): 1.36
Pipeline: Supporting APIs (workflow-support): 2.14
LDAP Plugin (ldap): 1.16
Pipeline: API (workflow-api): 2.20
JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1
ruby-runtime (ruby-runtime): 0.12
SSH Slaves plugin (ssh-slaves): 1.21
Pipeline: Basic Steps (workflow-basic-steps): 2.6
Script Security Plugin (script-security): 1.33
Structs Plugin (structs): 1.10
Windows Slaves Plugin (windows-slaves): 1.3.1
Translation Assistance plugin (translation): 1.15
Pipeline: Stage Step (pipeline-stage-step): 2.2
Credentials Plugin (credentials): 2.1.14
Authentication Tokens API Plugin (authentication-tokens): 1.3
JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1
Timestamper (timestamper): 1.8.8
Credentials Binding Plugin (credentials-binding): 1.13
GIT server Plugin (git-server): 1.7
Extended Choice Parameter Plug-In (extended-choice-parameter): 0.76
Plain Credentials Plugin (plain-credentials): 1.4
jQuery plugin (jquery): 1.11.2-0
CloudBees Amazon Web Services Credentials Plugin (aws-credentials): 1.21
Pipeline: Groovy (workflow-cps): 2.39
Durable Task Plugin (durable-task): 1.14
MapDB API Plugin (mapdb-api): 1.0.9.0
JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1
Run Condition Plugin (run-condition): 1.0
Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1
Scriptler (scriptler): 2.9
bouncycastle API Plugin (bouncycastle-api): 2.16.2
Git Changelog (git-changelog): 1.51
Pipeline: Step API (workflow-step-api): 2.12
Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.1.9
Pipeline: Model Definition (pipeline-model-definition): 1.1.9
Node Iterator API Plugin (node-iterator-api): 1.5
Job Configuration History Plugin (jobConfigHistory): 2.17
Display URL API (display-url-api): 2.0
Icon Shim Plugin (icon-shim): 2.0.3
Token Macro Plugin (token-macro): 2.2
Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.1.9
disk-usage plugin (disk-usage): 0.28
Pipeline: Job (workflow-job): 2.14.1
Environment Injector Plugin (envinject): 2.1.3
JUnit Plugin (junit): 1.21
Gerrit Trigger (gerrit-trigger): 2.25.0
Job DSL (job-dsl): 1.64
Ant Plugin (ant): 1.7
Matrix Authorization Strategy Plugin (matrix-auth): 1.7
Cobertura Plugin (cobertura): 1.11
Active Choices Plug-in (uno-choice): 1.5.3
Maven Integration plugin (maven-plugin): 2.17
SCM API Plugin (scm-api): 2.2.1
build timeout plugin (build-timeout): 1.18
HipChat Plugin (hipchat): 2.1.1
Jabber (XMPP) notifier and control plugin (jabber): 1.36
Mercurial plugin (mercurial): 2.0
Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.5
Hidden Parameter plugin (hidden-parameter): 0.0.4
Build Pipeline Plugin (build-pipeline-plugin): 1.5.7.1
Pipeline: Build Step (pipeline-build-step): 2.5.1
Discard Old Build plugin (discard-old-build): 1.05
CVS Plug-in (cvs): 2.13
Pipeline: Stage View Plugin (pipeline-stage-view): 2.8
Workspace Cleanup Plugin (ws-cleanup): 0.34
Mailer Plugin (mailer): 1.20
Resource Disposer Plugin (resource-disposer): 0.7
Matrix Project Plugin (matrix-project): 1.11
Pipeline: Input Step (pipeline-input-step): 2.8
Groovy Postbuild (groovy-postbuild): 2.3.1
Git client plugin (git-client): 2.5.0
Jackson 2 API Plugin (jackson2-api): 2.7.3
JaCoCo plugin (jacoco): 2.2.1
Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1
External Monitor Job Type Plugin (external-monitor-job): 1.7
Multiple SCMs plugin (multiple-scms): 0.6
Pipeline: Nodes and Processes (workflow-durable-task-step): 2.14
instant-messaging plugin (instant-messaging): 1.35
Copy Artifact Plugin (copyartifact): 1.38.1
Python Plugin (python): 1.3
Conditional BuildStep (conditional-buildstep): 1.3.6
SSH Credentials Plugin (ssh-credentials): 1.13
Pipeline: Model API (pipeline-model-api): 1.1.9
Configuration Slicing plugin (configurationslicing): 1.47
Hudson Post build task (postbuild-task): 1.8
Pipeline (workflow-aggregator): 2.5
Pipeline: SCM Step (workflow-scm-step): 2.6
Parameterized Trigger plugin (parameterized-trigger): 2.35.1
Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.8
OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.5
Pipeline: REST API Plugin (pipeline-rest-api): 2.8
Role-based Authorization Strategy (role-strategy): 2.5.1
Naginator (naginator): 1.17.2
Amazon Web Services SDK (aws-java-sdk): 1.11.119
EnvInject API Plugin (envinject-api): 1.2
Folders Plugin (cloudbees-folder): 6.1.2
Mask Passwords Plugin (mask-passwords): 2.10.1
Javadoc Plugin (javadoc): 1.4
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1
Git plugin (git): 3.5.1
Email Extension Plugin (email-ext): 2.58
Gitlab Hook Plugin (gitlab-hook): 1.4.2

 

Here is the config.xml

 

Here is the output from the console log after killing the job.

 
Started by timer
Obtained src/Jenkinsfile.CoconutPipeline.groovy from git git@git_server:namespace/job_repo.git
[Pipeline] node
Running on testops-w2 in /home/jenkins/workspace/Coconut/test-pipeline
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] hipchatSend*00:00:09.027* [INFO] HipChat notification sent to the following rooms: QA FW, Firmware, QA Automation
[Pipeline] stage
[Pipeline] { (Checkout)
[Pipeline] checkout*00:00:09.139* Cloning the remote Git repository*00:00:09.144* Cloning repository git@git_server:namespace/repo.git
Aborted by Username
[Pipeline] }
[Pipeline] // stage
[Pipeline] hipchatSend*01:26:16.377* [INFO] HipChat notification sent to the following rooms: QA FW, Firmware, QA Automation
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at hudson.remoting.Request.call(Request.java:147)
at hudson.remoting.Channel.call(Channel.java:838)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy101.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1075)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:260)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Finished: ABORTED
 

We try to keep Jenkins fairly up to date, and it's really annoying when this happens. It only happens on Pipeline jobs.

timja commented 7 years ago

tapsa:

This happen also in our CI environment:

Jenkins ver. 2.53

and plugins are:

cloudbees-folder6.1.2junit1.21ssh-credentials1.13script-security1.34matrix-project1.11antisamy-markup-formatter1.5github1.28.0pam-auth1.3git-client2.5.0windows-slaves1.3.1mailer1.20ldap1.17token-macro2.3mapdb-api1.0.9.0icon-shim2.0.3matrix-auth1.5pipeline-stage-step2.2external-monitor-job1.7subversion2.9build-timeout1.18credentials2.1.15ssh-slaves1.15structs1.10workflow-step-api2.13workflow-basic-steps2.6plain-credentials1.4credentials-binding1.13timestamper1.8.8email-ext2.59ws-cleanup0.34jira2.3ant1.7gradle1.27.1workflow-aggregator2.5jquery-detached1.2.1crowd21.8durable-task1.14workflow-api2.20workflow-support2.14monitoring1.69.1workflow-job2.10javadoc1.4pipeline-rest-api2.9handlebars1.1.1maven-plugin2.15.1momentjs1.1.1pipeline-stage-view2.9github-api1.86pipeline-build-step2.5.1jobConfigHistory2.18ace-editor1.1ssh-agent1.15workflow-scm-step2.4scm-api2.2.2workflow-cps2.40git-server1.7workflow-cps-global-lib2.9thinBackup1.9branch-api2.0.11git3.5.1workflow-multibranch2.16workflow-durable-task-step2.15pipeline-input-step2.8urltrigger0.41shelve-project-plugin1.5jquery1.11.2-1analysis-core1.92audit-trail2.2StashBranchParameter0.3.0job-import-plugin1.5parameterized-trigger2.35.2robot1.6.4rebuild1.25nested-view1.14claim2.9stashNotifier1.11.6envinject1.93.1mask-passwords2.10.1github-branch-source2.0.5radiatorviewplugin1.29display-url-api2.0analysis-collector1.52publish-over-ssh1.17xunit1.102job-node-stalker1.0.5run-condition1.0scp1.8msbuild1.27bouncycastle-api2.16.2git-parameter0.8.1cppcheck1.21config-file-provider2.16.4copyartifact1.38.1conditional-buildstep1.3.6jacoco2.2.1github-organization-folder1.6ivy1.27.1emma1.29build-user-vars-plugin1.5sonar2.6.1warnings4.63xvfb1.1.3embeddable-build-status1.9text-finder1.10pipeline-stage-tags-metadata1.1.9docker-commons1.8resource-disposer0.7docker-workflow1.13jenkins-flowdock-plugin1.1.8jackson2-api2.7.3pipeline-github-lib1.0stash-pullrequest-builder1.7.0pipeline-model-declarative-agent1.1.1pipeline-model-extensions1.1.4pipeline-graph-analysis1.5checkstyle3.49pipeline-milestone-step1.3.1pipeline-model-api1.1.9pipeline-model-definition1.1.4promoted-builds2.29.1build-name-setter1.6.7multiple-scms0.6valgrind0.27build-monitor-plugin1.12+build.201704111018authentication-tokens1.3sitemonitor0.5next-build-number1.4htmlpublisher1.14description-setter1.10

timja commented 7 years ago

jamesdumay:

Hi there,

Could you please generate and upload a support bundle using the support core plugin to provide further information here?

Thanks,
James

timja commented 7 years ago

michaelneale:

A support bundle https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin would be more useful - not sure anything can be reproduced from the above. 

timja commented 7 years ago

tapsa:

Add a support bundle support_2017-10-06_04.20.32.zip while several jobs are waiting SCM

timja commented 7 years ago

bparker:

Adding a support bundle for this issue. It is happening a lot today for some reason. 

support_2017-10-11_21.17.12.zip

 

Also, here is a stack trace from a hung checkout
Thread #4
at DSL.checkout(running in thread: org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution 4262)
at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar:file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:50)
at DSL.stage(Native Method)
at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar:file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:49)
at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.doCheckout(jar:file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:40)
at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/LabelScript.groovy:45)
at DSL.node(running on jenkins-w11 (jenkinsw-vm5))
at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/LabelScript.groovy:44)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:372)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:74)
at WorkflowScript.run(WorkflowScript:13)

timja commented 6 years ago

vrobert78:

Hi,

we have the same problem with a very updated version: 

Jenkins 2.89.3 (jenkins/jenkins:2.89.3-alpine)

GKE (k8s 1.8.7 - COS image)

 

ace-editor 1.1 true
antisamy-markup-formatter 1.5 true
apache-httpcomponents-client-4-api 4.5.3-2.1 true
authentication-tokens 1.3 true
blueocean 1.4.0 true
blueocean-autofavorite 1.2.1 true
blueocean-bitbucket-pipeline 1.4.0 true
blueocean-commons 1.4.0 true
blueocean-config 1.4.0 true
blueocean-core-js 1.4.0 true
blueocean-dashboard 1.4.0 true
blueocean-display-url 2.2.0 true
blueocean-events 1.4.0 true
blueocean-git-pipeline 1.4.0 true
blueocean-github-pipeline 1.4.0 true
blueocean-i18n 1.4.0 true
blueocean-jira 1.4.0 true
blueocean-jwt 1.4.0 true
blueocean-personalization 1.4.0 true
blueocean-pipeline-api-impl 1.4.0 true
blueocean-pipeline-editor 1.4.0 true
blueocean-pipeline-scm-api 1.4.0 true
blueocean-rest 1.4.0 true
blueocean-rest-impl 1.4.0 true
blueocean-web 1.4.0 true
bouncycastle-api 2.16.2 true
branch-api 2.0.18 true
build-user-vars-plugin 1.5 true
cloudbees-bitbucket-branch-source 2.2.9 true
cloudbees-folder 6.3 true
command-launcher 1.2 true
credentials 2.1.16 true
credentials-binding 1.15 true
display-url-api 2.2.0 true
docker-commons 1.11 true
docker-workflow 1.15 true
durable-task 1.17 true
favorite 2.3.1 true
git 3.7.0 true
git-client 2.7.1 true
git-server 1.7 true
github 1.29.0 true
github-api 1.90 true
github-branch-source 2.3.2 true
github-oauth 0.29 true
google-metadata-plugin 0.2 true
google-oauth-plugin 0.5 true
handlebars 1.1.1 true
handy-uri-templates-2-api 2.1.6-1.0 true
htmlpublisher 1.14 true
icon-shim 2.0.3 true
jackson2-api 2.8.10.1 true
javadoc 1.4 true
jenkins-design-language 1.4.0 true
jira 2.5.1 true
jquery 1.12.4-0 true
jquery-detached 1.2.1 true
jsch 0.1.54.1 true
junit 1.24 true
kubernetes 1.2 true
kubernetes-credentials 0.3.0 true
lockable-resources 2.1 true
mailer 1.20 true
matrix-auth 2.2 true
matrix-project 1.12 true
maven-plugin 3.1 true
mercurial 2.2 true
momentjs 1.1.1 true
oauth-credentials 0.3 true
pipeline-build-step 2.7 true
pipeline-github-lib 1.0 true
pipeline-graph-analysis 1.6 true
pipeline-input-step 2.8 true
pipeline-milestone-step 1.3.1 true
pipeline-model-api 1.2.7 true
pipeline-model-declarative-agent 1.1.1 true
pipeline-model-definition 1.2.7 true
pipeline-model-extensions 1.2.7 true
pipeline-rest-api 2.9 true
pipeline-stage-step 2.3 true
pipeline-stage-tags-metadata 1.2.7 true
pipeline-stage-view 2.9 true
plain-credentials 1.4 true
pubsub-light 1.12 true
scm-api 2.2.6 true
script-security 1.40 true
slack 2.3 true
sonar 2.6.1 true
sse-gateway 1.15 true
ssh-agent 1.15 true
ssh-credentials 1.13 true
ssh-slaves 1.25.1 true
structs 1.13 true
token-macro 2.3 true
variant 1.1 true
windows-slaves 1.3.1 true
workflow-aggregator 2.5 true
workflow-api 2.25 true
workflow-basic-steps 2.6 true
workflow-cps 2.44 true
workflow-cps-global-lib 2.9 true
workflow-durable-task-step 2.18 true
workflow-job 2.17 true
workflow-multibranch 2.17 true
workflow-scm-step 2.6 true
workflow-step-api 2.14 true
workflow-support 2.18 true

 

The build come in a infinite loop just after the "git rev-list".

We have to kill the build to stop it.

The problem appears sporadically, in general a replay successed.

The problem has appeared in the last 3 weeks.

 
Push event to branch dev15:11:08 Connecting to https://api.github.com using xxxx/******Obtained Jenkinsfile from xxxx
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] lock
Trying to acquire lock on [xxxx]
Lock acquired on [xxxx]
[Pipeline] {
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
Waiting for next available executor on jenkins-slave-w9k9c-5dghv
Running on
jenkins-slave-w9k9c-zh83m
in /home/jenkins/workspace/xxxx-L44IHDFACBXJ3M2RTOZ4YDAKTOFVJNRK4LJKN7AORR6T4YS5VNEQ
[Pipeline] {
[Pipeline] checkout
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository
https://github.com/xxx/xxx.git
> git init /home/jenkins/workspace/xxxx-L44IHDFACBXJ3M2RTOZ4YDAKTOFVJNRK4LJKN7AORR6T4YS5VNEQ # timeout=10
Fetching upstream changes from
https://github.com/ xxx/xxx .git
> git --version # timeout=10
using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress
https://github.com/ xxx/xxx .git
+refs/heads/dev:refs/remotes/origin/dev
> git config remote.origin.url
https://github.com/ xxx/xxx .git

  1. timeout=10
    > git config --add remote.origin.fetch +refs/heads/dev:refs/remotes/origin/dev # timeout=10
    > git config remote.origin.url
    https://github.com/ xxx/xxx .git
  2. timeout=10
    Fetching without tags
    Fetching upstream changes from
    https://github.com/ xxx/xxx .git
    using GIT_ASKPASS to set credentials
    > git fetch --no-tags --progress
    https://github.com/ xxx/xxx .git
    +refs/heads/dev:refs/remotes/origin/dev
    Checking out Revision xxxx (dev)
    > git config core.sparsecheckout # timeout=10
    > git checkout -f xxxx
    Commit message: "Merge tag 'xxxx' into dev"
    > git rev-list --no-walk xxxx # timeout=10
timja commented 6 years ago

michaelneale:

vrobert78 I see the same thing - git rev-list too... I haev a support core bundle for it too - taking a look

timja commented 6 years ago

michaelneale:

My support bundle support_2018-02-07_22.54.25.zip

 Slave-05 is the one that was hanging at the time if that helps. 

Hey markwaite any ideas on this? This seems to be a recently popped up thing, and it is biting more people (perhaps I should close and re-open this ticket, as the original one doesn't seem to be quite the same). 

timja commented 6 years ago

michaelneale:

could this be the same as https://issues.jenkins-ci.org/browse/JENKINS-46503 - in which case the solution is to use JIRA plugin 2.5.0 (not 2.5.1) - still not totally resolved. 

timja commented 6 years ago

vrobert78:

Hi michaelneale

i just disabled JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean. 

We will see during the day if the problem persists or not.

We will keep you in touch.

 

Best regards,

timja commented 6 years ago

markewaite:

vrobert78 I don't think it is enough to disable the Jira plugin. I believe that you need to downgrade the Jira plugin from 2.5.1 to 2.5.0 or earlier. There was a discussion of "blacklisting" Jira plugin 2.5.1 and that has been implemented about a week ago in the Jenkins update center.

Refer to JENKINS-48357 for the details of the types of damage done to critical libraries by the 2.5.1 release.

timja commented 6 years ago

vrobert78:

markewaite , since disabling the three plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean), hangs have gone. Everything has running fine yesterday.

timja commented 6 years ago

vrobert78:

We have just completly uninstall plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean) and then reinstall them.

Indeed, versions has changed:

We hope this will be as fine as yesterday.

timja commented 6 years ago

markewaite:

vrobert78 thanks very much for your investigation and for sharing your results. I'd like to close this bug as a duplicate of JENKINS-48357. JENKINS-48357 will remain open until the fix is released in the Jira plugin. Is that OK?

timja commented 6 years ago

vgaidarji:

svanoort ping

timja commented 2 years ago

[Duplicates: JENKINS-48357]