[ci] Next generation

[diafour]

Preflight Checklist

• [X] I agree to follow the Code of Conduct that this project adheres to.
• [X] I have searched the issue tracker for an issue that matches the one I want to file, without success.

Use case. Why is this important?

Migrating to GitHub Actions gave the project a step forward to openness. The first version of workflows has taught us how to work with GHA, it has lightened some problems with UI, and today we have some great ideas about our release processes.

This issue keeps problems and ideas in one place to discuss a new version of workflows.

Proposed Solution

• [ ] 1. Use comments in PRs to start additional workflows: e2e, documentation deploy, rebuild, validation, etc.

• [ ] 2. Fix e2e tests for external PRs.

• [ ] 3. Remove inlined javascript (github-script actions).

  • Move js to separate actions (e.g. https://github.com/grafana/grafana-github-actions for inspiration)

• [ ] 4. Remove inlined shell scripts.

  • Move building and testing instructions to Makefile

• [ ] 5. Find the alternative solution for external PRs

  • [ ] No way to use the main repo for testing workflow changes.
  • [ ] Build-and-push is mandatory for tests now, so secrets are needed.
  • [ ] pull_request trigger cons: "Workflows will not run on pull_request activity if the pull request has a merge conflict. The merge conflict must be resolved first."
  • [ ] pull_request_target trigger cons: "workflows are checked out from the base commit".

• [ ] 6. Bump versions for all actions.

• [ ] 7. Idea: move CI configuration to the separate repository

  • [ ] Example: https://github.com/kubernetes-sigs/release-utils
  • [ ] Idea: go templates are not handy, use cuelang to generate workflows.
  • [ ] Idea: use git subtree to include .github/workflows.

• [ ] 8. Make BOaTswain more interactive:

  • [ ] Comment PRs with possible commands.
  • [ ] Idea: use prow for slash commands (See kubernetes, kubermetes-sigs, kubevirt, openshift, etc.)

• [ ] 9. GHA emergency action plan.

  • [ ] Problem: we can't cut a new release when GHA is down.
  • [ ] Idea: add instructions to cut a new release without GHA.

• [ ] 10. Create cache cleaner for standalone runners.

• [ ] 11. Migrate js and shell scripts to use $GITHUB_ENV instead deprecated ::set-output

• [ ] 12. Add software to runners.

  • See list for cloud runner on Ubuntu20.04
  • Create instruction to launch new runner instance
  • Use ansible to install everything, see role

• [ ] 13. Use RUNNER_TEMP, drop /mnt/clout-layouts. See comment

• [ ] 14. Use shorter names for workflows and jobs

  

• [ ] 15. It is not possible to indicate actual trigger to workflow_dispatch'ed runs. There is a recommendation on GHA forums to use additional webapp for this purpose. For example: e2e run may be triggered from the PR, or from the release issue for tag, for release branch or for release branches with switching. All these types are displayed on the screenshot below 😅

  

• [ ] 16. Link to the milestone not guaranteed that commit is in the release branch. See https://github.com/deckhouse/deckhouse/pull/3109 — it was merged without the label status/backport. We need to check that milestone version is a separate branch and warn or even require status/backport label on PR.

• [ ] 17. Canceling e2e workflow run may lead to "upload" action starts archiving .git directory. It is not needed and just a waste of time. See https://github.com/deckhouse/deckhouse-test-2/actions/runs/3572436905/jobs/6005325478

  ...
  ##[debug]File: 645/8079. /home/ubuntu/actions-runner-test-2-1/_work/deckhouse-test-2/deckhouse-test-2/.git/objects/47/62764047bfa9918731808dac192ba58709d241 took 250.308 milliseconds to finish upload
  ##[debug]/home/ubuntu/actions-runner-test-2-1/_work/deckhouse-test-2/deckhouse-test-2/.git/objects/47/a1ed6cb96816d6d56560ab9babcea979630c9d is less than 64k in size. Creating a gzip file in-memory to potentially reduce the upload size
  ##[debug]A gzip file created for /home/ubuntu/actions-runner-test-2-1/_work/deckhouse-test-2/deckhouse-test-2/.git/objects/47/a1ed6cb96816d6d56560ab9babcea979630c9d helped with reducing the size of the original file. The file will be uploaded using gzip.
  ##[debug]File: 646/8079. /home/ubuntu/actions-runner-test-2-1/_work/deckhouse-test-2/deckhouse-test-2/.git/objects/47/8cdd878f8e3c2f487c810e001cae7bdbd95bb8 took 262.953 milliseconds to finish upload
  ##[debug]/home/ubuntu/actions-runner-test-2-1/_work/deckhouse-test-2/deckhouse-test-2/.git/objects/47/afbdf26e6519327ed2e1e301072bcc8ccb3700 is less than 64k in size. Creating a gzip file in-memory to potentially reduce the upload size
  ##[debug]The gzip file created for /home/ubuntu/actions-runner-test-2-1/_work/deckhouse-test-2/deckhouse-test-2/.git/objects/47/afbdf26e6519327ed2e1e301072bcc8ccb3700 did not help with reducing the size of the file. The original file will be uploaded as-is
  ##[debug]File: 647/8079. /home/ubuntu/actions-runner-test-2-1/_work/deckhouse-test-2/deckhouse-test-2/.git/objects/47/afbdf26e6519327ed2e1e301072bcc8ccb3700 took 271.764 milliseconds to finish upload
  ##[debug]/home/ubuntu/actions-runner-test-2-1/_work/deckhouse-test-2/deckhouse-test-2/.git/objects/47/b4dbf9d6d91b187cd29f696e556c93f7ff309c is less than 64k in size. Creating a gzip file in-memory to potentially reduce the upload size
  ##[debug]The gzip file created for /home/ubuntu/actions-runner-test-2-1/_work/deckhouse-test-2/deckhouse-test-2/.git/objects/47/b4dbf9d6d91b187cd29f696e556c93f7ff309c did not help with reducing the size of the file. The original file will be uploaded as-is
  ...

• [ ] 18. No informations about used images in comments. It is not possible to detect which job for switchin release and which is a regular run.

  

Additional Information

No response

[EvgenySamoylov]

Blocked by #1729