[JENKINS-50191] Multibranch Pipeline tends to keep rebuilding same commits

[timja]

We have a number of Pipeline jobs set up with automatic generation through Organization folders (both Stash and Github) and MultiBranch Pipelines generated from those (and finally PR/branch leaf jobs wherever Jenkinsfile's are found). Our default pipeline template includes an SCM polling setup (ranging from H/2 to H/15) because we don't have webhook ability due to corporate firewalling.

Quite often I see builds of the same commit IDs running in parallel, or even the later build starting hours after an earlier one has finished (even if successfully). They often have different causes, such as "Started by an SCM change" vs. "Branch indexing".

If a branch or PR has errors building (e.g. bad timing - new commit noticed in the original branch for a PR, but PR merged or closed during the half a minute or so while a build is preparing, so code can not be checked out - but MBP re-scanning did not yet happen to disable the job), then we can have hundreds of such failed rebuild attempts.

During FOSDEM-2018 discussion with abayer and markewaite (I'm not sure if we logged a ticket at that time) we concluded that such behavior is a bug, and that Jenkins is supposed to track in some matter which source commits it had processed (or even began processing - has queued or is building) and should avoid such automated duplicate re-runs of the same workload, regardless of whether the workspace (on an agent or master) is persistent or removed after each build.

This does not happen often (e.g. every time), but sufficiently frequently to be noticeable and annoying (and sometimes expensive - e.g. if this kicks off the avalanche of product build and re-testing that costs several hours of blocked resources here and there).

---

Originally reported by jimklimov, imported from: Multibranch Pipeline tends to keep rebuilding same commits

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[timja]

danielbeck:

Checkout needs to have finished for a build to be able to make subsequent SCM operations consider the commit already handled. Is this the problem? Huge checkouts, or perhaps even checkouts that aren't happening at the start of the build?

[timja]

apgray:

Seeing the same behaviour on Windows and latest LTS release 2.138.2 (however have seen the same behaviour since at least 2.107.x).

Our situation:

Multibranch pipeline (via a pollSCM(* * * * *) in the Jenkinsfile) will detect an SCM change (size of commit is irrelevant) and kick off a build.  Within a minute another build will be kicked off under the auspices of "Branch Indexing" (via the "Scan Multibranch Pipeline Triggers" setting).

This second duplicate "Branch Indexing build" is buggy behaviour.  The pollSCM renders the branch Indexing build redundant.  Messy workaround is to manually cancel second duplicate build.

[timja]

nikhilnanal:

I am facing the same issue with a multibranch pipeline (Jenkins 2.164, git hub branch source plugin 2.5.3, git plugin 3.10.0, scm -api 2.4.1). I am using scm polling enabled via the job configuration. If a PR build fails then it gets restarted arbitrarily during a subsequent polling interval even though there was no new change or pull request opened. These restarted PR builds hog my executors and if  a subsequent new PR comes in it has to wait  for the existing builds to complete or I have to manually cancel them. This is a serious problem for us. 

I am using "Merging the pull request with current target branch revision" option in my builds. could this be happening if a PR failed but other PRs passed which caused the target revision to be updated and hence the failed PR also gets retriggered?

[timja]

jimklimov:

For the bit about PR-merge builds, yes this is expected to re-run (once per seen target branch update though!) and part of the design idea about such ephemeral merge branches.

[timja]

jaygeorgewtc:

we are also facing similar issues. where an old build is kicked off of a stale branch via branch indexing where no changes is made with a same commit. which means the pipeline is deploying old code which causes huge problems for the dev team.  which is close to what other people are experiencing. 

Our versions 

Jenkins 2.222.3

git hub branch source plugin  2.6.0

git hub api  1.111

scm-api  2.6.3