Create MP Workflow Status Dashboard on Observability Platform

[richgreen-moj]

User Story

As a MP engineer I want to create a dashboard that monitors the success rate of our MP workflows (and create alerts if deemed useful) So that I can monitor any upward trends in failure rate or average running time etc.

Value / Purpose

• This would involve creating a dashboard that uses the ministryofjustice-github datasource available in the Observability Platform to create a table that displays the workflow-usage stats for all the MP workflows.
• This can be transformed to get the most useful data from the plugin so it is easy to see trends in the failure rate.
• We could consider some alerting if certain thresholds are met but initially it may only be useful as a quick overview to the health of our deployments.

Useful Contacts

@richgreen-moj

Additional Information

No response

Proposal / Unknowns

• The plugin currently only displays information for all branches but I have submitted a request to the developers to be able to specify main branch only as this would be more useful.
• Currently this will just be a manual process as we don't have the facility to implement via terraform on the OP, but this could be retro-fitted later.

Definition of Done

• [x] New dashboard created that shows stats for all MP workflows
• [x] Consider if any alerting would be useful with the team
• [x] Demo to the team - show how to navigate to dashboard, supply a link in documentation

[richgreen-moj]

I've created a dashboard which gives an overview of all the workflow runs in the MP repo: https://g-9d213fbc19.grafana-workspace.eu-west-2.amazonaws.com/d/l9nvX5wSk/modernisation-platform-workflow-stats?orgId=1

There are some pros and cons to the plugin and the information it provides:

Pros:

• A single pane of glass to view the health of all MP workflow runs over a period of time
• Ability to see average success/failure rate per workflow
• Ability to see the average running time for workflows
• Ability to add alerts based on the above metrics (e.g. set thresholds for failure rate or average time taken etc.)

Cons:

• Defaults to show stats for last 7 days, and when upping this to 30 days to get a months view the API times out/errors. There's an error on the 7-day view as certain workflows run on a monthly basis so there are no stats provided for the workflow.
• Stats are based on all branches rather than just main so the results are skewed while we carry out development on feature branches ( I think this is a reason not to add alerting at present - I've raised a feature request to filter by main on the plugin )
• The format of the metrics breaks when you try to filter by certain columns e.g. average run duration - when you sort in descending order a value of 9m5s come above 41m35s
• Similarly some of the stats have been pre-formatted into calculations e.g. percentage of failures, successes etc. which means you don't get the raw data in the query so it's harder to do visualisations (another feature request has been raised for this)