jenkinsci / generic-webhook-trigger-plugin

Can receive any HTTP request, extract any values from JSON or XML and trigger a job with those values available as variables. Works with GitHub, GitLab, Bitbucket, Jira and many more.
https://plugins.jenkins.io/generic-webhook-trigger
410 stars 161 forks source link

Filtering the branch name, triggers all or none branches pipelines #190

Closed tsboris closed 3 years ago

tsboris commented 3 years ago

Plugin version used: 1.72 Jenkins version used: 2.249.1 Git Repo: BitBucket Cloud

Git Structure

BitBucket Cloud repository with four branches:

Configuration

pipeline {
  agent { ecs { ... } }
  triggers {
    GenericTrigger(
      causeString: 'devops_test_repo Trigger',
      genericVariables: [
        [key: 'TRIGGER_BRANCH', value: '$.push.changes[0].new.name'],
      ],
      printContributedVariables: true,
      printPostContent: false,
      regexpFilterText: '$TRIGGER_BRANCH',
      regexpFilterExpression: '^master$|^develop$',
      token: 'devops_test_repo',
    )
  }
  stages {
    stage('TEST') {
      steps {
        echo "TEST"
      }
    }
  }
}

Build job log

Below, there are two build job logs.
The build logs are for when I triggered the master branch.
The first is for branch develop-test.
The second is for branch master.

Build job log for branch develop-test

[2020-11-25T07:53:22.569Z] devops_test_repo Trigger
[2020-11-25T07:53:22.827Z] Obtained Jenkinsfile from fa2c5f1842e90bf14fc879e7d9bb65ffb2588943
[2020-11-25T07:53:22.827Z] Running in Durability level: MAX_SURVIVABILITY
[2020-11-25T07:53:22.875Z] [Pipeline] Start of Pipeline
[2020-11-25T07:53:23.381Z] Contributing variables:
[2020-11-25T07:53:23.381Z] 
[2020-11-25T07:53:23.381Z]     TRIGGER_BRANCH = master
[2020-11-25T07:53:23.381Z] 
[2020-11-25T07:53:23.381Z] 
[2020-11-25T07:53:23.433Z] [Pipeline] ecsTaskTemplate
[2020-11-25T07:53:23.679Z] [Pipeline] {
[2020-11-25T07:53:23.816Z] Contributing variables:
[2020-11-25T07:53:23.817Z] 
[2020-11-25T07:53:23.817Z]     TRIGGER_BRANCH = master
[2020-11-25T07:53:23.817Z] 
[2020-11-25T07:53:23.817Z] 
[2020-11-25T07:53:23.961Z] [Pipeline] node
[2020-11-25T07:53:38.966Z] Still waiting to schedule task
[2020-11-25T07:53:38.966Z] All nodes of label ‘develop-test-5-8xh91’ are offline
[2020-11-25T07:53:49.575Z] Running on AWSNodeCluster-develop-test-5-8xh91-xvxpl in /home/jenkins/workspace/test_job_Pipeline_develop-test
[2020-11-25T07:53:49.578Z] [Pipeline] {
[2020-11-25T07:53:49.590Z] Contributing variables:
[2020-11-25T07:53:49.590Z] 
[2020-11-25T07:53:49.590Z]     TRIGGER_BRANCH = master
[2020-11-25T07:53:49.590Z] 
[2020-11-25T07:53:49.590Z] 
[2020-11-25T07:53:49.594Z] [Pipeline] stage
[2020-11-25T07:53:49.597Z] [Pipeline] { (Declarative: Checkout SCM)
[2020-11-25T07:53:49.609Z] Contributing variables:
[2020-11-25T07:53:49.609Z] 
[2020-11-25T07:53:49.609Z]     TRIGGER_BRANCH = master
[2020-11-25T07:53:49.609Z] 
[2020-11-25T07:53:49.609Z] 
[2020-11-25T07:53:49.621Z] [Pipeline] checkout
[2020-11-25T07:53:49.625Z] Contributing variables:
[2020-11-25T07:53:49.625Z] 
[2020-11-25T07:53:49.625Z]     TRIGGER_BRANCH = master
[2020-11-25T07:53:49.625Z] 
[2020-11-25T07:53:49.625Z] 
[2020-11-25T07:53:49.833Z] The recommended git tool is: NONE
[2020-11-25T07:53:52.770Z] using credential atlassian_account
[2020-11-25T07:53:52.776Z] Contributing variables:
[2020-11-25T07:53:52.776Z] 
[2020-11-25T07:53:52.776Z]     TRIGGER_BRANCH = master
[2020-11-25T07:53:52.776Z] 
[2020-11-25T07:53:52.776Z] 
[2020-11-25T07:53:52.782Z] Cloning the remote Git repository
[2020-11-25T07:53:52.782Z] Cloning with configured refspecs honoured and without tags
[2020-11-25T07:53:52.819Z] Cloning repository https://bitbucket.org/acme_repo/devops_test_repo.git
[2020-11-25T07:53:52.881Z]  > git init /home/jenkins/workspace/test_job_Pipeline_develop-test # timeout=10
[2020-11-25T07:53:52.921Z] Fetching upstream changes from https://bitbucket.org/acme_repo/devops_test_repo.git
[2020-11-25T07:53:52.922Z]  > git --version # timeout=10
[2020-11-25T07:53:52.937Z]  > git --version # 'git version 2.26.2'
[2020-11-25T07:53:52.937Z] using GIT_ASKPASS to set credentials atlassian_account
[2020-11-25T07:53:52.939Z]  > git fetch --no-tags --force --progress -- https://bitbucket.org/acme_repo/devops_test_repo.git +refs/heads/develop-test:refs/remotes/origin/develop-test # timeout=10
[2020-11-25T07:53:53.822Z] Avoid second fetch
[2020-11-25T07:53:53.822Z] Contributing variables:
[2020-11-25T07:53:53.822Z] 
[2020-11-25T07:53:53.822Z]     TRIGGER_BRANCH = master
[2020-11-25T07:53:53.822Z] 
[2020-11-25T07:53:53.822Z] 
[2020-11-25T07:53:53.823Z] Checking out Revision fa2c5f1842e90bf14fc879e7d9bb65ffb2588943 (develop-test)
[2020-11-25T07:53:53.796Z]  > git config remote.origin.url https://bitbucket.org/acme_repo/devops_test_repo.git # timeout=10
[2020-11-25T07:53:53.809Z]  > git config --add remote.origin.fetch +refs/heads/develop-test:refs/remotes/origin/develop-test # timeout=10
[2020-11-25T07:53:53.833Z]  > git config core.sparsecheckout # timeout=10
[2020-11-25T07:53:53.846Z]  > git checkout -f fa2c5f1842e90bf14fc879e7d9bb65ffb2588943 # timeout=10
[2020-11-25T07:53:57.563Z] Commit message: "Testing Webhook"
[2020-11-25T07:53:57.569Z]  > git rev-list --no-walk e55fe7b892c5591c0390ca480f3091eb2fafcd5a # timeout=10
[2020-11-25T07:53:57.627Z] [Bitbucket] Notifying commit build result
[2020-11-25T07:53:57.780Z] [Pipeline] }
[2020-11-25T07:53:57.790Z] [Pipeline] // stage
[2020-11-25T07:53:57.800Z] Contributing variables:
[2020-11-25T07:53:57.800Z] 
[2020-11-25T07:53:57.800Z]     TRIGGER_BRANCH = master
[2020-11-25T07:53:57.800Z] 
[2020-11-25T07:53:57.800Z] 
[2020-11-25T07:53:57.806Z] [Pipeline] withEnv
[2020-11-25T07:53:57.811Z] [Pipeline] {
[2020-11-25T07:53:57.825Z] Contributing variables:
[2020-11-25T07:53:57.825Z] 
[2020-11-25T07:53:57.825Z]     TRIGGER_BRANCH = master
[2020-11-25T07:53:57.825Z] 
[2020-11-25T07:53:57.825Z] 
[2020-11-25T07:53:57.830Z] [Pipeline] stage
[2020-11-25T07:53:57.832Z] [Pipeline] { (TEST)
[2020-11-25T07:53:57.858Z] Contributing variables:
[2020-11-25T07:53:57.858Z] 
[2020-11-25T07:53:57.858Z]     TRIGGER_BRANCH = master
[2020-11-25T07:53:57.858Z] 
[2020-11-25T07:53:57.858Z] 
[2020-11-25T07:53:57.864Z] [Pipeline] echo
[2020-11-25T07:53:57.866Z] TEST
[2020-11-25T07:53:57.871Z] [Pipeline] }
[2020-11-25T07:53:57.883Z] [Pipeline] // stage
[2020-11-25T07:53:57.890Z] [Pipeline] }
[2020-11-25T07:53:57.899Z] [Pipeline] // withEnv
[2020-11-25T07:53:57.906Z] [Pipeline] }
[2020-11-25T07:53:57.964Z] [Pipeline] // node
[2020-11-25T07:53:57.971Z] [Pipeline] }
[2020-11-25T07:53:58.149Z] [Pipeline] // ecsTaskTemplate
[2020-11-25T07:53:58.155Z] [Pipeline] End of Pipeline
[2020-11-25T07:53:58.161Z] [Bitbucket] Notifying commit build result
[2020-11-25T07:53:58.289Z] [Bitbucket] Build result notified
[2020-11-25T07:53:58.296Z] Finished: SUCCESS

Build job log for branch master

[2020-11-25T07:53:22.572Z] devops_test_repo Trigger
[2020-11-25T07:53:22.808Z] Obtained Jenkinsfile from 7b602d6ca56612f16b15eee573a8879d53c77ad3
[2020-11-25T07:53:22.808Z] Running in Durability level: MAX_SURVIVABILITY
[2020-11-25T07:53:22.863Z] [Pipeline] Start of Pipeline
[2020-11-25T07:53:23.287Z] Contributing variables:
[2020-11-25T07:53:23.287Z] 
[2020-11-25T07:53:23.287Z]     TRIGGER_BRANCH = master
[2020-11-25T07:53:23.287Z] 
[2020-11-25T07:53:23.287Z] 
[2020-11-25T07:53:23.320Z] [Pipeline] ecsTaskTemplate
[2020-11-25T07:53:23.577Z] [Pipeline] {
[2020-11-25T07:53:23.789Z] Contributing variables:
[2020-11-25T07:53:23.790Z] 
[2020-11-25T07:53:23.790Z]     TRIGGER_BRANCH = master
[2020-11-25T07:53:23.790Z] 
[2020-11-25T07:53:23.790Z] 
[2020-11-25T07:53:23.815Z] [Pipeline] node
[2020-11-25T07:53:38.962Z] Still waiting to schedule task
[2020-11-25T07:53:38.962Z] All nodes of label ‘master-34-8vz9q’ are offline
[2020-11-25T07:54:08.834Z] Running on AWSNodeCluster-master-34-8vz9q-rr0rj in /home/jenkins/workspace/test_job_Pipeline_master
[2020-11-25T07:54:08.838Z] [Pipeline] {
[2020-11-25T07:54:08.852Z] Contributing variables:
[2020-11-25T07:54:08.852Z] 
[2020-11-25T07:54:08.852Z]     TRIGGER_BRANCH = master
[2020-11-25T07:54:08.852Z] 
[2020-11-25T07:54:08.852Z] 
[2020-11-25T07:54:08.856Z] [Pipeline] stage
[2020-11-25T07:54:08.859Z] [Pipeline] { (Declarative: Checkout SCM)
[2020-11-25T07:54:08.871Z] Contributing variables:
[2020-11-25T07:54:08.871Z] 
[2020-11-25T07:54:08.871Z]     TRIGGER_BRANCH = master
[2020-11-25T07:54:08.871Z] 
[2020-11-25T07:54:08.871Z] 
[2020-11-25T07:54:08.882Z] [Pipeline] checkout
[2020-11-25T07:54:08.887Z] Contributing variables:
[2020-11-25T07:54:08.887Z] 
[2020-11-25T07:54:08.887Z]     TRIGGER_BRANCH = master
[2020-11-25T07:54:08.887Z] 
[2020-11-25T07:54:08.887Z] 
[2020-11-25T07:54:09.151Z] The recommended git tool is: NONE
[2020-11-25T07:54:12.294Z] using credential atlassian_account
[2020-11-25T07:54:12.298Z] Contributing variables:
[2020-11-25T07:54:12.298Z] 
[2020-11-25T07:54:12.298Z]     TRIGGER_BRANCH = master
[2020-11-25T07:54:12.298Z] 
[2020-11-25T07:54:12.298Z] 
[2020-11-25T07:54:12.302Z] Cloning the remote Git repository
[2020-11-25T07:54:12.302Z] Cloning with configured refspecs honoured and without tags
[2020-11-25T07:54:12.333Z] Cloning repository https://bitbucket.org/acme_repo/devops_test_repo.git
[2020-11-25T07:54:12.412Z]  > git init /home/jenkins/workspace/test_job_Pipeline_master # timeout=10
[2020-11-25T07:54:12.558Z] Fetching upstream changes from https://bitbucket.org/acme_repo/devops_test_repo.git
[2020-11-25T07:54:12.559Z]  > git --version # timeout=10
[2020-11-25T07:54:12.584Z]  > git --version # 'git version 2.26.2'
[2020-11-25T07:54:12.584Z] using GIT_ASKPASS to set credentials atlassian_account
[2020-11-25T07:54:12.586Z]  > git fetch --no-tags --force --progress -- https://bitbucket.org/acme_repo/devops_test_repo.git +refs/heads/master:refs/remotes/origin/master # timeout=10
[2020-11-25T07:54:13.306Z] Avoid second fetch
[2020-11-25T07:54:13.307Z] Contributing variables:
[2020-11-25T07:54:13.307Z] 
[2020-11-25T07:54:13.307Z]     TRIGGER_BRANCH = master
[2020-11-25T07:54:13.307Z] 
[2020-11-25T07:54:13.307Z] 
[2020-11-25T07:54:13.307Z] Checking out Revision 7b602d6ca56612f16b15eee573a8879d53c77ad3 (master)
[2020-11-25T07:54:13.276Z]  > git config remote.origin.url https://bitbucket.org/acme_repo/devops_test_repo.git # timeout=10
[2020-11-25T07:54:13.289Z]  > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10
[2020-11-25T07:54:13.318Z]  > git config core.sparsecheckout # timeout=10
[2020-11-25T07:54:13.333Z]  > git checkout -f 7b602d6ca56612f16b15eee573a8879d53c77ad3 # timeout=10
[2020-11-25T07:54:17.036Z] Commit message: "Testing Webhook"
[2020-11-25T07:54:17.040Z]  > git rev-list --no-walk ade01aa86fa6ee7aa9139e83baa8b4f77a57261f # timeout=10
[2020-11-25T07:54:17.105Z] [Bitbucket] Notifying commit build result
[2020-11-25T07:54:17.219Z] [Pipeline] }
[2020-11-25T07:54:17.229Z] [Pipeline] // stage
[2020-11-25T07:54:17.238Z] Contributing variables:
[2020-11-25T07:54:17.238Z] 
[2020-11-25T07:54:17.238Z]     TRIGGER_BRANCH = master
[2020-11-25T07:54:17.238Z] 
[2020-11-25T07:54:17.238Z] 
[2020-11-25T07:54:17.243Z] [Pipeline] withEnv
[2020-11-25T07:54:17.246Z] [Pipeline] {
[2020-11-25T07:54:17.261Z] Contributing variables:
[2020-11-25T07:54:17.261Z] 
[2020-11-25T07:54:17.261Z]     TRIGGER_BRANCH = master
[2020-11-25T07:54:17.261Z] 
[2020-11-25T07:54:17.261Z] 
[2020-11-25T07:54:17.265Z] [Pipeline] stage
[2020-11-25T07:54:17.268Z] [Pipeline] { (TEST)
[2020-11-25T07:54:17.296Z] Contributing variables:
[2020-11-25T07:54:17.296Z] 
[2020-11-25T07:54:17.296Z]     TRIGGER_BRANCH = master
[2020-11-25T07:54:17.296Z] 
[2020-11-25T07:54:17.296Z] 
[2020-11-25T07:54:17.301Z] [Pipeline] echo
[2020-11-25T07:54:17.304Z] TEST
[2020-11-25T07:54:17.309Z] [Pipeline] }
[2020-11-25T07:54:17.322Z] [Pipeline] // stage
[2020-11-25T07:54:17.329Z] [Pipeline] }
[2020-11-25T07:54:17.338Z] [Pipeline] // withEnv
[2020-11-25T07:54:17.344Z] [Pipeline] }
[2020-11-25T07:54:17.405Z] [Pipeline] // node
[2020-11-25T07:54:17.418Z] [Pipeline] }
[2020-11-25T07:54:17.532Z] [Pipeline] // ecsTaskTemplate
[2020-11-25T07:54:17.538Z] [Pipeline] End of Pipeline
[2020-11-25T07:54:17.545Z] [Bitbucket] Notifying commit build result
[2020-11-25T07:54:17.649Z] [Bitbucket] Build result notified
[2020-11-25T07:54:17.656Z] Finished: SUCCESS

Response from Jenkins to BitBucket

Below, there are two responses.
The first is when triggering branch develop-test.
The second is when triggering branch master.

Response when triggering branch develop-test

{
  "jobs": {
    "test_job_Pipeline/develop-test": {
      "regexpFilterExpression": "^master$|^develop$",
      "triggered": false,
      "resolvedVariables": {
        "TRIGGER_BRANCH": "develop-test"
      },
      "regexpFilterText": "develop-test",
      "id": 0,
      "url": ""
    },
    "test_job_Pipeline/master": {
      "regexpFilterExpression": "^master$|^develop$",
      "triggered": false,
      "resolvedVariables": {
        "TRIGGER_BRANCH": "develop-test"
      },
      "regexpFilterText": "develop-test",
      "id": 0,
      "url": ""
    },
    "test_job_Pipeline/develop-test-12": {
      "regexpFilterExpression": "^master$|^develop$",
      "triggered": false,
      "resolvedVariables": {
        "TRIGGER_BRANCH": "develop-test"
      },
      "regexpFilterText": "develop-test",
      "id": 0,
      "url": ""
    },
    "test_job_Pipeline/develop": {
      "regexpFilterExpression": "^master$|^develop$",
      "triggered": false,
      "resolvedVariables": {
        "TRIGGER_BRANCH": "develop-test"
      },
      "regexpFilterText": "develop-test",
      "id": 0,
      "url": ""
    }
  },
  "message": "Triggered jobs."
}

Response when triggering branch master

{
  "jobs": {
    "test_job_Pipeline/develop-test": {
      "regexpFilterExpression": "^master$|^develop$",
      "triggered": true,
      "resolvedVariables": {
        "TRIGGER_BRANCH": "master"
      },
      "regexpFilterText": "master",
      "id": 1490,
      "url": "queue/item/1490/"
    },
    "test_job_Pipeline/master": {
      "regexpFilterExpression": "^master$|^develop$",
      "triggered": true,
      "resolvedVariables": {
        "TRIGGER_BRANCH": "master"
      },
      "regexpFilterText": "master",
      "id": 1492,
      "url": "queue/item/1492/"
    },
    "test_job_Pipeline/develop-test-12": {
      "regexpFilterExpression": "^master$|^develop$",
      "triggered": true,
      "resolvedVariables": {
        "TRIGGER_BRANCH": "master"
      },
      "regexpFilterText": "master",
      "id": 1491,
      "url": "queue/item/1491/"
    },
    "test_job_Pipeline/develop": {
      "regexpFilterExpression": "^master$|^develop$",
      "triggered": true,
      "resolvedVariables": {
        "TRIGGER_BRANCH": "master"
      },
      "regexpFilterText": "master",
      "id": 1489,
      "url": "queue/item/1489/"
    }
  },
  "message": "Triggered jobs."
}

Expected result and actual result

Case 1: Pushing a commit into branch develop-test. Expected: Jenkins shouldn't run any build jobs at all. Actual: Jenkins is not running any build jobs

Case 2: Pushing a commit into branch master. Expected: Jenkins should run a single build job only for branch master. Actual: Jenkins is running build jobs for each branch, even those that are not the master branch (that’s four simultaneous job runs).

tsboris commented 3 years ago

This is a multibranch Pipeline

tomasbjerre commented 3 years ago

The readme contains:

...
     regexpFilterText: '$ref',
     regexpFilterExpression: 'refs/heads/' + BRANCH_NAME
...

Which makes the job trigger a build when $ref resolved to the branch corresponding to the job created by Multibranch.

Perhaps this will work for you:

pipeline {
  agent { ecs { ... } }
  triggers {
    GenericTrigger(
      causeString: 'devops_test_repo Trigger',
      genericVariables: [
        [key: 'TRIGGER_BRANCH', value: '$.push.changes[0].new.name'],
      ],
      printContributedVariables: true,
      printPostContent: false,
      regexpFilterText: '$TRIGGER_BRANCH $TRIGGER_BRANCH',
      regexpFilterExpression: '^' + BRANCH_NAME + ' master$|develop$',
      token: 'devops_test_repo',
    )
  }
  stages {
    stage('TEST') {
      steps {
        echo "TEST"
      }
    }
  }
}
tsboris commented 3 years ago

Hi Tomas, Your suggestion did not resolve the issue :-(

For the purpose of further investigation, I've reduced the BitBucket Cloud repository to just two branches: master and develop-test-12 After several more testing using a very simple filter, my conclusions are as follows.

Trigger setup:

triggers {
    GenericTrigger(
      causeString: 'devops_test_repo Trigger',
      genericVariables: [
        [key: 'TRIGGER_BRANCH', value: '$.push.changes[0].new.name'],
      ],
      printContributedVariables: true,
      printPostContent: false,
      regexpFilterText: '$TRIGGER_BRANCH',
      regexpFilterExpression: 'master',
      token: 'devops_test_repo',
    )
  }

Based on the above basic and simple setup, only branch master should trigger a Jenkins build and the build should only run in master branch pipeline

What actually happens is:

Response when triggering branch develop-test-12

{
  "jobs": {
    "test_job_Pipeline/master": {
      "regexpFilterExpression": "master",
      "triggered": false,
      "resolvedVariables": {
        "TRIGGER_BRANCH": "develop-test-12"
      },
      "regexpFilterText": "develop-test-12",
      "id": 0,
      "url": ""
    },
    "test_job_Pipeline/develop-test-12": {
      "regexpFilterExpression": "master",
      "triggered": false,
      "resolvedVariables": {
        "TRIGGER_BRANCH": "develop-test-12"
      },
      "regexpFilterText": "develop-test-12",
      "id": 0,
      "url": ""
    }
  },
  "message": "Triggered jobs."
}

Please notice that when the trigger is develop-test-12 branch, the value of "triggered": false is set as false for both test_job_Pipeline/master and test_job_Pipeline/develop-test-12.
This is expected because the trigger is NOT master, so all branches should be set as "triggered": false, "id": 0 and "url": ""

Response when triggering branch master

{
  "jobs": {
    "test_job_Pipeline/master": {
      "regexpFilterExpression": "master",
      "triggered": true,
      "resolvedVariables": {
        "TRIGGER_BRANCH": "master"
      },
      "regexpFilterText": "master",
      "id": 1818,
      "url": "queue/item/1818/"
    },
    "test_job_Pipeline/develop-test-12": {
      "regexpFilterExpression": "master",
      "triggered": true,
      "resolvedVariables": {
        "TRIGGER_BRANCH": "master"
      },
      "regexpFilterText": "master",
      "id": 1817,
      "url": "queue/item/1817/"
    }
  },
  "message": "Triggered jobs."
}

Please notice that when the trigger is master branch, the value of "triggered": true is set as true for both test_job_Pipeline/master and test_job_Pipeline/develop-test-12.
I would expect it to be:

tomasbjerre commented 3 years ago

This is expected with that filter. And the fix is what I suggested in previous response:

...
      regexpFilterText: '$TRIGGER_BRANCH $TRIGGER_BRANCH',
      regexpFilterExpression: '^' + BRANCH_NAME + ' master$|develop$',
...

Because the multibranch will create 2 jobs. The first part '^' + BRANCH_NAME will make sure these jobs only trigger on changes to the branch they were created for. The second part + ' master$|develop$' makes sure it only triggers for the branch that was changed.

You can view the applied configuration of the jobs created by multibranch and you will see that BRANCH_NAME is resolved upon creation of the jobs.

The master job will have expression '^master master$|develop$'. The develop job will have expression '^develop master$|develop$'.

tsboris commented 3 years ago

Hi Tomas, Your fix is working. Thanks!

bharathpantala commented 1 year ago

how to pass the branch to branch field to clone