FIREBREAK: 🔥 Create a Dashboard For Displaying KPIs

[connormaglynn]

Background

We have a lot of automated jobs in the operations-engineering repository. These jobs were built to solve a problem. We have very little visibility over whether these jobs actually still solve that initial problem except for when we have to react to a symptom.

For an example, we have a job to configure standards on a repository via a label. This job runs everyday, although we don't have an easy way to determine if it provides any value besides manually checking the logs everyday.

Questions / Assumptions

• Will storing time series KPIs of systems and processes help to:
  • proactively decommission low-value systems?
  • highlight high-value systems?
• Can we use Promethues/Grafana for this?
• Can we use the Observabiity Platform for this?
• Can we use Cloud Platform for this?
• Can we build our own custom database and dashboard UI for this?

What hypothesis do we want to test?/What do we want to learn?

We want to learn the following:

• Is it possible to export KPIs from GitHub Actions to monitor the value of a process
• Does doing this provide insight and value

Peripheral benefits of doing this:

• more familiarity with observability tools such as Promethues, Grafana, Time Series databases (all this depends on the solution we chose to use)

Definition of done

• [x] Firebreak finding documented appropriately
• [x] Demo completed
• [ ] Decision made on whether to progress Firebreak work
• [x] A dashboard displaying KPIs for at least one production process

[connormaglynn]

To add a bit of commentary - this is a large idea with multiple approaches for a solution! ✅

Ideally, use this problem as an excuse to research a technology stack that interests you (that may also solve the problem 🙈 ) 🚀

[connormaglynn]

High-Level Overview of solution architecture 👇

The solution will consist of three components:

1. KPI Dashboard - the UI interface that stakeholders will primarily interact will to view and analyse KPI metrics i.e. view line graphs, bar charts etc.
2. KPI Database - a database to store simple time series data of KPIs i.e. number of repositories using the "standards" label over time. Used to populate the KPI Dashboard
3. Exporter - exporters will sit on clients and push data to the KPI Database (probably through an API). An exporter/push method of gathering metrics is preferred since the primary data source will be ephemeral (i.e. cron jobs that don't live long enough to be scraped).

[connormaglynn]

Custom Implementation

🆙 Update

• For the dashboarding UI, I've chosen to use the Dash - a low-code Python framework for building data apps
• For the data source, I've chosen PostgresDB a SQL database that I'm familiar with
• For the exporter, I've decided to leave it out, for now, and stub the data manually. This is because exposing an endpoint to populate data into a database would require security practices before being pushed to a live environment. Leaving out this technical requirement, for now, allows the UI and database to be deployed live safely.
• Database Schema

  CREATE TABLE IF NOT EXISTS indicators (
      id SERIAL PRIMARY KEY,
      indicator varchar,
      timestamp timestamp,
      count integer
  );

• High-level overview of current implementation using custom components 👇
• Snapshot of what the current Dash dashboard looks like, integrating with a PostgresDB (with manually populated data) 👇

👀 Remaining Work

• Decide whether to invest more into the custom dashboard route i.e. deploy it live, create an exporter or stress test the complexity of data that can be held by PostgresDB and effectively displayed by Dash OR to move onto using different components for the solution i.e. an actual time series database and a more comprehensive dashboarding tool

📝 Notes

• Exporter Security Concerns - To deploy the exporter live into the primary data source (GitHub Actions) the API to populate the database will need to be available via the public internet. Due to this, an implementation to authenticate the GitHub action will also need to be implemented. This requirement is a technical complexity that I believe will distract from proving the value of the system as a whole. I have therefore chosen to not implement it and instead invest the time in better understanding the constraints of the data stores/dashboarding tools

[connormaglynn]

Custom Implementation

🆙 Update

• I've added some additional graphs to the dashboard which displayed metrics around our GitHub Actions usage and Support Stats. The source data was in CSV format which Plotly supported fairly easily! 🥳
• POC for the custom implementation using Postgres and Dash/Plotly can be cloned at operations-engineering-kpi-dashboard-poc

👀 Remaining Work

• Am going to move on to testing how Grafana integrates with CSV and Postgres data formats - since these seem like likely candidates for storing data

📝 Notes

• Worth noting that GitHub in some cases only support extracting data via CSV, so a KPI Dashboard solution would do well to support CSV formats for dashboarding

[connormaglynn]

Grafana

🆙 Update

• Added a Grafana dashboard which pulls data from both a Postgres Database and CSV files
• Grafana runs locally as part of a docker-compose file

👀 Remaining Work

• The live data of these services KPIs seems valuable enough to justify capturing the data, so I will stop experimenting with different components and start to build a live Postgres Database with a secure API to populate live data into 🚀
• Need to experiment separately with time series databases such as Prometheus, InfluxDB etc.

📝 Notes

• Grafana made it easy to build dashboard UIs, but I struggled with Wide Format data. I was unable to successfully convert the Support Stats data into a suitable format for Grafana itself.
• Grafana also didn't offer easy tools to add additional annotations such as a trend line for the data, which Dash/Plotly offers out of the box
• Breifly looking at Prometheus showed that there are potentially some more verbose querying methods that would allow extracting additive data such as the mean, min, max etc. Which may work better with Grafana, since Grafana doesn't offer this functionality easily.

[connormaglynn]

🆙 Update

• Created a live app and database to display stubbed data at https://kpi-dashboard-poc.cloud-platform.service.justice.gov.uk/

👀 Remaining Work

• Add an API to allow a secure method of populating the live database

[connormaglynn]

🆙 Update

• [x] Demo shown to to the team
• [x] Dashboard now automatically tracks one live production metric of the number of repositories that have the standards-compliant label
• [x] Final architecture looks like

👀 Remaining Work

• [ ] Decision to carry on with dashboard