Last failed release pipeline on trunk branch(stag/qa whatever) block subsequent PR pipeline on same branch from getting merged

[nagenderRawat]

We have a use case where we have a qa branch(consider it as a main branch) and run CI steps for every PR made from diff feature branches(cut from qa and introduce additional changes) to qa and once those feature => qa PR are merged we use to run release pipeline for qa which does the CD stuff such that code is deployed on qa cluster. Problem arise when a release pipeline got failed(for any reason) and causing lighthouse not to auto merge all subsequent PR's made on qa branch.

The steps that leads to reproducing issue are as follows -:

1. A developer cut/create a feature/x branch from qa.
2. Make change as per requirement of feature in the feature/x branch and raise PR 101 to qa branch from feature/x (source branch).
3. As a result lighthouse triggers pr pullrequest pipeline(i.e a presubmits ) for PR 101, which basically does the CI(lint/test/sq etc) checks and if passes, merges feature/x to qa branch.
4. Now the qa branch is having the new changes, so release pipeline runs, which builds the image and deploy it to qa cluster.

Now let suppose the release pipeline got failed(step 4) for some reason.

Now if next developer wants to make a PR to qa for his own feature branch i.e feature/y, than this new PR from feature/y => qa will not be automatically merged because of the last release pipeline failed on qa branch and last release pipeline status getting pulled on the PR checks of this latest PR(can be seen in screenshot).

As a result subsequent PR's are not getting auto merged by lighthouse on qa branch once a release pipeline is failed on qa branch.

.lighthouse/jenkins-x/triggers.yaml looks like this

apiVersion: config.lighthouse.jenkins-x.io/v1alpha1
kind: TriggerConfig
spec:
  presubmits:
  - name: pr
    context: "pr"
    always_run: true
    optional: false
    trigger: "/test"
    rerun_command: "/retest"
    source: "pullrequest.yaml"
    branches:
    - qa
  postsubmits:
  - name: release
    context: "release"
    source: "release.yaml"
    branches:
    - qa

lighthouse version -:

- chart: jenkins-x/lighthouse
  version: 0.0.851
  name: lighthouse
  namespace: jx
  values:
  - versionStream/charts/jenkins-x/lighthouse/values.yaml.gotmpl

Using latest jenkins x 3 with tekton catalog.

Please refer screenshot

[nagenderRawat]

@jstrachan

[jstrachan]

as I said on the call - you should not configure presubmits to use the qa branch; since they are not triggered from the qa branch. The release pipeline should never be triggered on a PR at all so should never show up on a PR checks page.

I really don't understand whats going on - but its worth remembering the branches are patterns; are you creating a PR on a branch called feature/qa or something which is why the PR triggers a PR and release?

[nagenderRawat]

No, release pipeline is not called for any PR.

Let me explain it in a diff way...

When I(admin on repo) directly push a commit on qa branch a release pipeline(let suppose release pipeline name is qa release #1) gets triggered(as i configuration trigger.yaml to do so). And for instance qa release #1 got failed due to some obvious reasons.

Now, if a next person(A)who is not authorised to push a commit directly to qa branch want's to merge his feature code to qa branch. Then the only option A is left with is to work on a feature branch(let suppose feature/migration) which is cut from qa branch...

Now, when A makes PR(No. 124, let suppose ) from it's feature/migration branch to qa branch a pr pipeline gets triggered (as trigger.yaml is also config to run pullrequest pipeline when PR is made to qa branch from any branch) but the checks on this particular PR is also showing the last failed release pipeline (i.e qa release #1) along with this currently running pr pipeline....

Note -: No new release pipeline is called for PR 124, the release pipeline which you see in screenshot is one that ran for the last release made on qa(qa release #1), when commit was directly pushed to qa branch(explained above), which got failed at that time...and due to this last failed release pipeline(qa release #1) status pulled over on this current PR the lighthouse automerge is affected...

In total the main issue is why the last release pipeline is showing on current active PR 124's checks...!

[nagenderRawat]

It might now happen for your repos coz your pr/github process doesn't require to run pr and release pipeline both for same branch... But our's does have that...as we use to run CI steps through pr pipeline whenever PR is made on qa branch from any feature branch (i.efeature=>qa) and once that particular PR is merge to qa branch we use to run release pipeline which pick that latest qa branch code and build/deploy it to relevant cluster... That's why you will see qa branch specified in both pre and post summits in trigger.yaml...

[jenkins-x-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle stale

[jenkins-x-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle rotten