Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra

jenkins-infra / helpdesk

Open your Infrastructure related issues here for the Jenkins project

https://github.com/jenkins-infra/helpdesk/issues/new/choose

16 stars 10 forks source link

Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra #2778

Open dduportal opened 2 years ago

dduportal commented 2 years ago

No more GitHub actions for updatecli (less configuration spread)
- For instance https://github.com/jenkins-infra/jenkins-infra/pull/2059
- Full control of the execution environment of updatecli
Updatecli pipeline triggers can be controlled separately from repository own pipelines
- Simplifies repository own pipelines (less conditions)
- Allow to fine tune updatecli frequency: weekly, daily, etc.
- repository own pipelines would be completely separated from the "chore" tasks of updatecli
Centralized credentials in infra.ci.jenkins.io (which is private)

Convention Proposal:

Use a Jenkinsfile_updatecli a pipeline marker
Define a new GH organization scanning on infra.ci.jenkins.io that will check ALL of jenkins-infra and will check for these updatecli processes
Write down a documentation of this in jenkins-infra/documentation

Current progress: https://github.com/jenkins-infra/helpdesk/issues/2778#issuecomment-2222690819

dduportal commented 2 years ago

Blocked by https://github.com/jenkins-infra/helpdesk/issues/2834 (because of involved credentials)

dduportal commented 1 year ago

Status:

PoC nice on kubernetes-management

Next step:

As it's per-repo basis, we have to implement GH org. scanning (wip on helm chart)
Slightly minor issue: the 2 jobs have the same GH check status. How to change that?

lemeurherve commented 5 months ago

We're planning to do one per week, with azure & packer-images for this one.

dduportal commented 5 months ago

Update:

Project jenkins-infra/azure ✅
- https://github.com/jenkins-infra/kubernetes-management/pull/4954
- https://github.com/jenkins-infra/azure/pull/617
Project jenkins-infra/packer-images ✅
- https://github.com/jenkins-infra/kubernetes-management/pull/3855
- https://github.com/jenkins-infra/packer-images/pull/1039

dduportal commented 5 months ago

This issues causes merge problems as all jobs are sending a continuous-integration/jenkins/pr-merge check (along with the "custom name" we added for the new updatecli jobs.

It means the checks can be all green if the last job to run was green, despite the first one being red.

Example with https://github.com/jenkins-infra/packer-images/pull/1041 where the main (and most important) build fails but updatecli jobs succeeded after the main job failure: PR can be merged as checks are all marked jobs are green:

dduportal commented 5 months ago

@lemeurherve I'm not sure how to handle this unwanted behavior. Looks like it is related to Jenkins Multibranch pipeline not supporting multiple pipeline per repositories. Do you think we could generate a custom name for each of our project so we could have an explicit check for "main builds" too?

dduportal commented 5 months ago

After a first discussion, we started to look into fully switching to GitHub Status Check system (disabled by default in infra.ci unless explicitly set up, such as the Updatecli distinct jobs).

E.g. Defining up a default GitHub Status Check setup with the controller FDQN as "name", skipping status update and disabling all notifications (such as `continuous-integration/jenkins/pr-merge) emitted by the GitHub Branch Source Plugin. Unless a custom configuration is provided for status check of course.

Changing

=> That would be a workable solution for now, involving "only" a major change on the jenkins-jobs helm chart. Only inconvenience would be the risk on accidentally leaking build log (it is the default for now but can be enabled accidentally through a bad setup).

After another discussion with other contributors, it appears that we missed https://plugins.jenkins.io/github-branch-source/#plugin-content-extension-plugins:

github-scm-trait-notification-context - allows overriding the continuous-integration/jenkins/ commit status name.

A first quick check shows that this plugin could help us:

Capture d’écran 2024-02-21 à 17 15 47

gave us (look at the non Required notification)

Capture d’écran 2024-02-21 à 17 16 18

No more dependency (and risk of build output leaks) on the status checks plugin
No more "2 clicks from a PR to reach Jenkins"
Deterministic behavior and name
No "complex" configuration on the JobDSL jenkins-jobs chart (and no breaking change)

dduportal commented 5 months ago

Proposal in https://github.com/jenkins-infra/helm-charts/pull/1081

lemeurherve commented 4 months ago

Updating repositories' required status checks in branch protection one by one when working on them, don't hesitate to ping here or on #jenkins-infra like https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$sQD_zXduwJpkJ9IAHnlO_UiGTYbTXeI2qcvm-QeFa58 if there is a particular repo PR stuck with a pending required status check.

lemeurherve commented 2 weeks ago

Update:

azure-net:
- https://github.com/jenkins-infra/azure-net/pull/273
- https://github.com/jenkins-infra/kubernetes-management/pull/5413

lemeurherve commented 2 weeks ago

Current progress on jenkins-infra repositories containing an updatecli folder:

All docker-* repositories are using the parallelDockerUpdatecli, to be splitted?

lemeurherve commented 2 weeks ago

Note: https://github.com/jenkins-infra/docker-jenkins-weekly & https://github.com/jenkins-infra/docker-jenkins-infraci will be taken care of in #4171

dduportal commented 2 weeks ago

All docker-* repositories are using the parallelDockerUpdatecli, to be splitted?

Absolutely!

lemeurherve commented 2 weeks ago

While they were opened, I updated my PRs to use https://github.com/apps/jenkins-infra-updatecli instead of https://github.com/jenkins-infra-bot in updatecli values. (No more co-authored commits with both updatecli GitHub app and jenkins-infra-bot GitHub bot account)