Automate application release process

[laurencejackson]

It would be useful to automate as much of the release process as possible.

The objectives for the release pipeline:

• Automate as much as possible
• Ensure all artifacts are logged and tracked
• Majority of release processing will be on Github actions to enable visibility and automation

After a discussion with @AnilMistry, we have drafted the below solution. This proposes a new function in the csc-mlops CLI release that would be used to trigger the release pipeline, after this function has run all processes are automatic assuming no errors.

mlops release <path_to_mlflow_tracked_run: string> <type_of_release: major,minor,patch>

This function will collect all required artifacts associated with the mlflow run and push them to a cloud location, in the short term this will probably be a LFS enabled github repository. A series of github actions will be added to each repository that will perform the following tasks:

1. run unit tests
2. build MAP package
3. build documentation

A final task will then build a final application using the MAP as a base image and copy in the documentation and other relevant build artifacts.

This is a proposed automated release pipeline, any feedback would be appreciated @GSTT-CSC/core.

[dangerdika]

@laurencejackson This looks really cool and comprehensive to me but I dont know enough for my input to be valuable here - would you be able to find the time to explain the flow diagram at a 10x instead?

[heyhaleema]

@laurencejackson I like this! What happens to some of the build artefacts e.g., the unit test results, if the process stops before the task that uses them e.g., the documentation, is never achieved?

[laurencejackson]

Thanks! Since all of the build artefacts are generated within a GitHub action, if the action fails then they will be lost when that action terminates. I don't think we'll need any artefacts from failed builds, but if we did I dont think this is a problem since they can always be reproduced using the same commands on the same commit.

The logic for the GitHub action will require all steps to succeed before publishing the final package