Multibranch Pipeline triggering broken

[medianick]

I'm suspecting that #305 may have broken our ability to trigger Multibranch Pipeline jobs via the Generic Webhook Trigger. These are jobs with a triggers section in their Jenkinsfile like this:

    triggers {
      GenericTrigger(
        causeString: 'Label applied',
        genericVariables: [
          [defaultValue: '', key: 'TRIGGER_ACTION', value: '$.action'],
          [defaultValue: '', key: 'LABEL_NAME', value: '$.label.name']
        ],
        token: 'some_token'
      )
   }

It appears that after our upgrade from 2.0.2 to 2.2.0, new MultiBranch Pipeline jobs -- i.e., individual branch/PR jobs created (via branch indexing) by a Multibranch Pipeline after this plugin upgrade was applied -- are no longer found. This means that the invoke?token=some_token webhook deliveries for MBP jobs created entirely after this upgrade fail (HTTP 404) with this:

{
    "jobs": null,
    "message": "Did not find any jobs with GenericTrigger configured!\nA token was supplied.\n"
}

And webhook deliveries for MBP jobs that already existed prior to the upgrade only list the older jobs in the webhook response -- i.e., those branches/PRs that existed prior to the upgrade -- and the newer jobs are not triggered at all.

This seems to be the result of either 2.2.0 or 2.2.1. We've been running 2.2.0 since April 26th and 2.2.1 since May 23rd.

[tomasbjerre]

Probably same problem as #311 and #312. Solution is to upgrade Jenkins.

I'll probably make a new release with this new version of Jenkins as required, there are several issues like this one lately.

[medianick]

Indeed, upgrading to Jenkins 2.452.2 has resolved the issue. Thanks for the guidance, @tomasbjerre!

[medianick]

Actually, this is improved (with 2.452.2 installed), but not totally resolved.

The original branch I was testing with in my Multibranch Pipeline job is now getting detected (and is visible in the response payload when calling the webhook). So, that's the improvement, and it works perfectly for that original branch.

However, I created a new branch -- which the MBP picked up via the usual branch indexing, so it exists in Jenkins -- but triggering the webhook (either via GitHub, or manually) does not list that branch in the response; it still only shows the original branch. As a result, actions that should be triggered by this webhook for this new branch are not triggered at all.

[medianick]

To provide a little more background, my setup is a little complex:

• Webhook added to a repo to call a GWT URL with a repo-specific token
• Multibranch Pipeline job (created manually), using a particular Jenkinsfile name in the repo in question
• Jenkinsfile invokes a shared library function that defines a pipeline with the desired GenericTrigger setup in it
• The token configured in the GenericTrigger is a parameter for the shared library function

[medianick]

I've also tried creating an entirely new MBP job (just copying from the existing one) but the GWT webhook isn't finding that either. So, it's basically as if the only job and branch that works is the one that existed at the point we upgraded to 2.452.2 (and Jenkins was restarted).

[tomasbjerre]

It works on restart because this plugin will gather all jobs on startup: https://github.com/jenkinsci/generic-webhook-trigger-plugin/blob/3954ee99f8c8801877f6a9b998fe7fdcaa78e6d7/src/main/java/org/jenkinsci/plugins/gwt/jobfinder/JobFinderImpersonater.java#L60

It is a problem with the updating.

[medianick]

It is a problem with the updating.

Is that likely to be a problem in Jenkins itself or in this plugin?

[tomasbjerre]

It is a problem with the updating.

Is that likely to be a problem in Jenkins itself or in this plugin?

Or maby multibranch pipeline plugin.

[medianick]

Upon further testing, it's possible this isn't specific to Multibranch Pipelines. I created a simple Job DSL script that seeds a (regular, not Multibranch) Pipeline job with GenericTrigger in its (inline, not from SCM) script, and the GWT webhook URL is not finding that seeded job either. E.g.:

pipelineJob('TestJob') {
  definition {
    cps {
      sandbox(true)
      script("""
pipeline {
    agent none
    triggers {
        GenericTrigger(
... etc.

Possibly it will find it the next time our Jenkins instance is restarted.

[medianick]

I also tried copying a manually created Pipeline job that uses GenericTrigger (which is working fine) to a new job, and the new copy was found (listed in the webhook response) but not triggered ("triggered":false) until I opened and re-saved its configuration.

[tomasbjerre]

I asked for advice here: https://groups.google.com/g/jenkinsci-dev/c/w-jdCAtGh20

[mawinter69]

One thing to consider for pipeline jobs is that anything that affects the job configuration and is set via the Jenkinsfile becomes only visible after the pipeline has run at least once (also any change will be effective only after the next run). So the behaviour that a job created with JobDSL containing a declarative pipeline script which defines the trigger it is expected that the job will not be triggered by the hook. If you start the job manually once, then next time the hook is called the job should be triggered. Jenkins will not scan a Jenkinsfile upon creation of the job for any configured triggers.

Multibranch is triggering a first run upon job creation unless disabled. So if you disabled the first run this would explain the behaviour you see.

[medianick]

The multibranch pipeline behavior we were relying on previously was that branch indexing would automatically trigger the build of a newly discovered branch (PR), which would register the trigger as expected, such that a subsequent operation -- in our case, adding a label to a pull request -- would then allow the GWT webhook call to trigger that branch build (and indeed, that of any other branches still active with that same GWT token registered). Within the Jenkinsfile in question, we conditionally excluded stages (via when blocks) based on actual incoming parameters so that only the intended branch and expected GitHub operation would actually cause the "real" build steps to run. This multibranch pipeline job was created via Job DSL.

This is what has stopped working as of 2.2.0 or 2.2.1 (unless it was an unrelated change to a different plugin updated around the same time).

[tomasbjerre]

When multibranch scans the repo, I would expect an invocation of:

• onCreated with triggers
• or onCreated without triggers, followed by an onUpdated with triggers

It is very strange if none of those occurred because the plugin is obviously configured in the job. Visible in the GUI and also available when getting the jobs the old way.

If none of those two alternatives are fulfilled, I think this plugin cannot use itemListener to keep track of the jobs. It would need to revert back to the slow implementation from #305 .

[mawinter69]

Looking at https://github.com/jenkinsci/branch-api-plugin and https://github.com/jenkinsci/workflow-multibranch-plugin they don't seem to call an ItemListener. So it might be necessary to implement a SaveableListener instead or in parallel (to cover job moves)

[tomasbjerre]

I implemented SaveableListener: https://github.com/jenkinsci/generic-webhook-trigger-plugin/commit/291833b906bc8ab4dc8320b122f3a3d575320732

Releasing that now as 2.2.2.

[medianick]

Fantastic! Thanks, @tomasbjerre. I'll report back once we upgrade and I can test it.

[medianick]

Just confirming that 2.2.2 has solved the issue. We installed that on Friday night and have tested successfully this morning.