A holistic approach to versioning, CICD and audit reporting

[nithyashree675]

A suggested workflow to enhance marge-bot's scope in order to cover the complete landscape of software delivery, by enabling her take care of -

1. Versioning: by configuring gitlabCI pipelines to formulate a suitable tag for this version of the release artifact (basically container image tag) as part of https://github.com/smarkets/marge-bot/issues/341
2. CI: Current functionality validates that the master is green. As future work, run a job to push the images with the tags chosen in [1] onto a container registry (to be configured in gitlabCI, ideally after merging into master)
3. CD: Update the tag from [1] in argoCD image.tag values configuration to deploy the latest image (preferably shipping to staging first, validate/test, promote to master)
4. Audit reporting: Being able to traceback all the steps from the MR was assigned to marge-bot and changes reflect in production is crucial. We already have reviewed-by field, probably we should have a component asset registrar or an centralised ops portal with notifications.

Extended features described above can make both developers and SRE team's work efficient not just by avoiding manual processes but also speeds up the cycle with confidence and integrity.

[bmistry12]

Please ensure that this proposed feature set is not too tailored towards Smarkets requirements and does consider the open-source community. A few initial thoughts come to mind:

1. Enabling gitOps and audit reporting should be flags, as not everyone may want to use this functionality.
2. It seems you are proposing these CI pipelines for adding versioning tags, what is the thought process for adding this requirement?
   • While tagging is useful for releasing and tracking versions of release changes, there may be certain parts of a codebase / particular repo's that you do not want to enable this feature for - e.g. if I am not building or pushing artefacts in the change.
   • Surely it makes more sense to tag when releases are made/cut rather than on every commit?
3. Marge should have the possibility to be compatible with a variety of CD tools and not just be restricted to ArgoCD.
4. CD is a difficult challenge and everyone implements this process differently - when do we autodeploy vs click-on-demand deploy etc. This probably is something that should be decoupled from Marge. Her primary focus surrounds CI, and unless there are plans to cater for all the different forms of CD / CD tools, adding this will muddy the water.

5. Component asset registrar or a centralised ops portal with notifications.

   • Adding git commits / details to the git log is a more flexible solution in that there is no dependency on external tools.
   • Like with CD if a choice is made to use an asset register, this should have the extensibility to support any other asset registries the community would like to use.
   • Notifications seem more like a CD requirement? What would be the requirement for centralised notifications for merged changes?

6. How are features 1/2/3 useful to me if I am not dealing with containerisation?
7. How does this workflow affect the run time of Marge? We still need to make sure changes can be merged at a reasonable rate, especially if urgent changes are being made.
   • A drastic breakdown to put my concern across, but: as you increase the Marge feature set, you're also increasing the final CI runtime, and the backlog of changes to be merged, and increasing the chances of batching (if enabled), this increases the chance of infinite batching, increasing the possibility of Marge being stuck in a sad loop of not merging anything 😁
8. A valid point is raised in the PR linked to the linked ticket - jenkins retry should be removed in favour of calling merge_request_pipeline_trigger.