For a single job, multiple builds are sometimes triggered for a single commit

[dhumeniuk]

On an intermittent basis, a single job will be started twice for a single change. When I push a commit to Stash, sometimes the same Jenkins job (for that development branch or maybe a job that runs off the master when merging to the master) will be started twice.

I do have "Execute concurrent builds if necessary" enabled for the job in Jenkins as I do want to have multiple builds running if there is a newer commit pushed while the existing build is still running.

[mikesir87]

This sounds more of a configuration issue with your Jenkins setup than a bug for this plugin. This webhook merely notifies Jenkins when commits and pull requests have been merged. What your Jenkins instance does from there is on you with how you have it configured.

With that preface, I'm going to go ahead and close this issue. However, I'd be happy to help you diagnose and resolve your issue. Feel free to continue the conversation.

[dhumeniuk]

How does the notification to Jenkins work. Is it some kind of web service request? If there are multiple commits from different branches (or even repos), would the same notification be sent to Jenkins? So maybe two different commits from two different branches are triggering the same job that is running from a single branch?

[dhumeniuk]

Here's the polling log from the 2 consecutive builds as an example:

Started on Apr 16, 2014 4:18:17 PM Polling SCM changes on ssgwin7slave Using strategy: Default [poll] Last Built Revision: Revision 17b3c680b3ad20a4c7cb989ea454869be7024a77 (origin/feature/TH-2336-Update-XXXX1.25) Fetching changes from the remote Git repositories Fetching upstream changes from https://xxx.yyy.com/stash/scm/th/ts14.git using .gitcredentials to set credentials Polling for changes in Done. Took 5 sec Changes found

Started on Apr 16, 2014 4:18:22 PM Polling SCM changes on ssgwin7slave Using strategy: Default [poll] Last Built Revision: Revision 17b3c680b3ad20a4c7cb989ea454869be7024a77 (origin/feature/TH-2336-Update-XXXX1.25) Fetching changes from the remote Git repositories Fetching upstream changes from https://xxx.yyy.com/stash/scm/th/ts14.git using .gitcredentials to set credentials Polling for changes in Done. Took 5.1 sec Changes found

[mikesir87]

Here's how the plugin works...

1) An update is made to a repository and an event is published. 2) The webhook plugin listens to various events, so is notified of the event. 3) The webhook plugin determines if a notification should be sent (based on settings for black/whitelisting of branches, committers, etc.) 4) If a notification is to be sent, a GET request is made to Jenkins using a URL that is made available by the Git Plugin.

This is where the webhook's plugin functionality ends. However, the Git Plugin then performs the following...

5) The Git Plugin takes the repository provided in the Git repository and determines if there is a project matching that repository. 6) The Git Plugin makes a poll request to see if there was an actual update made (by comparing the latest commit hash against the last one it knew about). 7) If there is a difference, the code is then updated in the local workspace and a build is triggered.

For more info on how the Git Plugin works, including the URL the webhook uses, see the project's wiki page at https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin

[dhumeniuk]

I assume this is the URL that you are speaking of in the link?

http://yourserver/jenkins/git/notifyCommit?url=[&branches=branch1[,branch2]*][&sha1=]

Will the branches and sha1 values be set by the webhook plug-in?

[mikesir87]

It is the URL. And no, the branches and sha1 values are not provided, since the Git Plugin determines the changes anyways.

[dhumeniuk]

Okay, so under the following circumstances:

Branches: master, feature1, feature2 Jobs: build-feature1 and build-feature2 (both restricted to their respective feature branch)

1. Commit is pushed to feature1 and feature2 near simultaneously
2. Webhook triggers Jenkins twice, once for each push
3. GitPlugin gets first trigger, checks all branches, GitPlugin determines there are changes in both feature1 and feature2, queues up the builds
4. GitPlugin gets 2nd trigger, checks all branches, GitPlugin determines there are changes in both feature1 and feature2, queues up the builds (since the build hasn't actually started)
5. feature1 and feature2 builds start from the first check
6. feature1 and feature2 builds start from the 2nd check

Seems like the GitPlugin could be changed, but it might depend on if the GitPlugin would know if there is another change when it polls the second time as it hasn't completed getting the change to see where the previous build will build to.

Maybe I'm missing something, but would it be possible for the webhook to be updated to fill in the branch property as well to avoid this race condition?

[chasepeeler]

In my instance, the hook should only notify for changes on the develop branch, and the only change made to the develop branch is the merging of the pull request. Unless stash, behind the scenes, does two updates as part of a pull request merge. Of course, that's assuming that putting the branch name in the "build only" setting causing the notification to only be sent for that single branch.

[chasepeeler]

I noticed this: Notified Stash for commit with id ec3fa352887840ea1e012fa75b1575cf2f98e1bd

I have a post build job to notify stash, but, that happened at the start of the build. After the git stuff, but before the first build task. Perhaps that is doing something that is causing stash to send the change notification to Jenkins. At that point, it's the same scenario as above, where it examines the branch, and, since the first build is still in progress, sees that there are changes.

[chasepeeler]

I removed the stash notification, and I'm still getting two builds for every merge.

[chasepeeler]

I've setup network monitor on my jenkins server, filtered to only show http traffic from our stash server. The next time we merge a pull request, I should be able to see if it's sending two notifications.

[chasepeeler]

Those are the requests made to the jenkins server from the stash server. 10:11 is when the pull request was merged, and there are two notifyCommits. About 40 seconds later, without anything having been done on against the repository, except for some (not sure how much) of the jenkins commands:

Started by an SCM change Building in workspace d:/workspace

git.exe rev-parse --is-inside-work-tree Fetching changes from the remote Git repository git.exe config remote.origin.url https://git.endeavortelecom.com/scm/ndvr/esp.git Fetching upstream changes from https://git.endeavortelecom.com/scm/ndvr/esp.git git.exe --version using .gitcredentials to set credentials git.exe config --local credential.helper store --file=\"C:\windows\TEMP\git1701435348473673097.credentials\" git.exe fetch --tags --progress https://git.endeavortelecom.com/scm/ndvr/esp.git +refs/heads/:refs/remotes/origin/ git.exe config --local --remove-section credential git.exe rev-parse "origin/develop^{commit}" Checking out Revision e5bfd103de7a9600ebfad55b0a180c75b4429287 (origin/develop) git.exe config core.sparsecheckout git.exe checkout -f e5bfd103de7a9600ebfad55b0a180c75b4429287 git.exe rev-list ea5a6e298215377cf092ae6619a087d173649243

It doesn't look to me like any of those should cause the jenkins hook to send another notifyCommit request.

[chasepeeler]

If I push to my develop branch via cli (instead of merging via pull request) it doesn't trigger multiple builds. I've also noticed that if I cancel the 2nd build quickly, while it's still queued up, it will trigger a third build. I have yet to see it trigger a 4th build.

[l8nite]

Did you guys ever work this out? I'm seeing double builds triggered for a single commit too.

[dhumeniuk]

No. Still having the issue.

Dave Humeniuk On Jun 25, 2014 1:32 AM, "Shaun Guth" notifications@github.com wrote:

Did you guys ever work this out? I'm seeing double builds triggered for a single commit too.

— Reply to this email directly or view it on GitHub https://github.com/Nerdwin15/stash-jenkins-postreceive-webhook/issues/63#issuecomment-47062119 .

[iginzburg]

I am having the same issue. Any updates?

[chasepeeler]

We worked around the issue by setting a quiet time - so, I'm not sure if it's still happening or not. I can't remove the quiet time and confirm right now.

[pavlop-asg]

The same for me

[chr0n1x]

bump

[crick231]

I see build getting triggered for 1. branch creation 2. git push 3. pull request creation 4. merge to master, is there anyway we can skip triggering build while branch creation?

[pkahn-attivio]

Hi Folks, We see this two and we noticed a pattern jenkins reports humans triggering push for first build and our build user triggering the second. We worked around the problem by placing a user filter on the trigger on our build user. The build user doesn't commit, it just makes tags.

Our theory (mythology really) is that because we tag as a part of our builds each build triggers another build due to the tag. We cant really rep it.

Our solution is to write a github payload interceptor similar to our scmsynchronizer to receive a payload from github and then only forward it if it isn't just a tag made by the build user or is code unrelated to source (say a change to systemtest which isn't executed in a compile job). Once complete, we may open source our playload interceptor provided legal allows it.

[victorv]

Same problem here. In our case we get two notifications in github that point to the same build but the status of one is "building" and the other is "complete"

[anthuwan13]

Was there any solution found for this issue , as we face the same issue now.

[curious-man]

I join. I have the same problem with merge develop in master

[saurabh-pathak94]

We are seeing builds being triggered twice for same commit notification if Branch Specifier is set to '**' (any). I am believe it is related to this issue.

[Redempter]

Hey, I am having the same issue with my Jenkins server, one commit, two builds being triggered for the same job. Has anyone found something for this?

[rzinnatullin]

Hi, I'm experiencing the same issue with Jenkins. However, this doesn't happen with TeamCity server. I think TeamCity can somehow filter duplicate triggerings.

[qizer]

Hi, Looks like this plugin not free now and this version out of support \=

I also have the same issue.

For me there two triggers:

1. Branch event
2. commit notification