[JENKINS-38443] Pollscm trigger runs the job twice for the same commit

[timja]

We've got the job with pollscm trigger enabled for two repositories.
The job is triggered twice with one minute interval, though both commit hashes are the same.

Pollscm logs look like this:

Started on Sep 22, 2016 9:47:00 AM
Using strategy: Default
[poll] Last Built Revision: Revision  (go/master)
...
[poll] Latest remote head revision on refs/heads/master is:  - already built by 1
Using strategy: Default
...
[poll] Latest remote head revision on refs/heads/master is: 
Done. Took 1 sec
Changes found

Started on Sep 22, 2016 9:48:00 AM
Using strategy: Default
[poll] Last Built Revision: Revision  (go/master)
...
[poll] Latest remote head revision on refs/heads/master is:  - already built by 2
Using strategy: Default
...
[poll] Latest remote head revision on refs/heads/master is: 
Done. Took 0.65 sec
Changes found

pair is the same in two job runs.

---

Originally reported by ifn, imported from: Pollscm trigger runs the job twice for the same commit

• status: Reopened
• priority: Critical
• resolution: Unresolved
• imported: 2022/01/10

[timja]

sonneveldsmartward:

When I had a pipeline job's PollSCM property configured with "@daily", I witnessed 7 builds triggered for the same changeset. All were "Started by an SCM change". Is it possible that PollSCM isn't checking for a running build with that changeset? Although why would it attempt to build more than once a day?

[timja]

ifn:

It seems that setting concurrent to false prevents from rerunning the job for the same commit.
I'm not sure that it is connected with Pipeline plugin.

[timja]

danielbeck:

You're probably polling too often. Jenkins looks at past builds to determine what has already been built, and if there's not enough time to start a build and checkout from SCM to record what the current state is, Jenkins will just queue another build, as what has been found hasn't been built yet.

[timja]

sonneveldsmartward:

As far as I know, we're only polling "daily" per branch, configured via pollSCM. We do have a mercurial hook that GETs "/mercurial/notifyCommit" for each commit but I feel that shouldn't affect things as that is just notifying jenkins that "something" changed. The parent multibranch job does not have any polling configured.

[timja]

ifn:

Jenkins will just queue another build, as what has been found hasn't been built yet

As far as I understand, queuing the job repeatedly for the same commit is ok. Strange though. Thanks.

[timja]

sonneveldsmartward:

Linking to related JENKINS-39092 . The difference is that no polling interval is set, only change notifications

[timja]

timblaktu:

We encountered this problem in a dramatic fashion last night in a branch of a multibranch pipeline job whose pipelines did NOT set disableConcurrentBuilds() option, in a jenkins cluster containing:

• 1 Controller (Zero Executors)
• 2 Build Agents (2 Executors Each)

A single branch job using `pollSCM('H/5 * * * *')` trigger entered a feedback look where every 5 minutes it was triggered (for the same revision) even though there were no SCM changes.

Reading danielbeck's comment, I see that the pollSCM mechanism didn't see that this specific revision was built because these builds were waiting for an executor for long periods of time, so the SCM polling triggers would keep happening even if there were no changes.

The executor stall can be considered the "root cause", and we can attempt to fix this with infrastrucutre, however, it would be nice to have a way to prevent pollSCM triggers from "running amok" in this way.

Yes, we could disableConcurrentBuilds() in all branch job pipelines, but pushing such a change to all branches is very cumbersome. It would be nice to be able to specify this in JobDSL. But as I discuss Here multibranch pipelines do not appear to support disabling Concurrent builds via jobDSL. As I describe, rate limiting is not appropriate here, since it requires the jobDSL to know something about build time. We want a simple "build serialization" that is provided by disableConcurrentBuilds in pipeline, without requiring every Jenkinsfile to specify it.

[timja]

JIRAUSER130555:

As I wrote here
probably it might help to poll the SCM only when there is a free executor ready to immediately take over the job to be started. Then we do not end up queueing up jobs indefinitely, however it might not avoid in all cases that a job is started twice for a change, but it should avoid that the pollSCM will run amok triggering more and more builds.

[timja]

JIRAUSER130555:

When there is a change pollSCM could also remember the state of the last polling, when it was "No changes" then a job may be even triggered when no executor is available as then it is sure that this is a fresh change, not yet with a build in progress.
Then no new jobs are triggered until the previously started job handled the change and brought the state back to "No changes".
This would also allow parallel builds as it is assured that the last jobs has checked out already before a new one gets triggered.

[timja]

[Originally related to: JENKINS-39092]