benthorner commented 4 years ago

As part of the work for https://github.com/alphagov/govuk-puppet/pull/9894 we identified a large number of Grafana dashboards that aren't in version control. Although this is not an issue with the govuk-puppet repo itself, this seems like the most relevant place to document them all and discuss what to do. Most of the dashboards can probably be deleted without much discussion, because they:

Have graphs with no data
Are completely empty
Are broken (show errors)
Are incomplete (lack titles, graphs, reusability, etc.)
Duplicate another dashboard

Dashboards to keep

The following dashboards seem to be working and look complete/useful. We should clarify who 'owns' them and if they are interested in adding them to version control.

Production

[x] "AWS second line" (working)
[x] "Assets" (working)
[x] "Fastly CDN" (working)
[x] "Icinga Service Status Overview (Chris B)" (working)
[x] "GDS API Adapters errors (Chris B)" (no data)

Production AWS

[x] "Brexit Readiness" (working)
[x] "Icinga Service Status Overview (Chris B)" (working)
[x] "Search requests and errors" (working)
[x] "TCP Connections by Machine (Chris Baines)" (working)

Staging

[x] "Assets" (working)

Staging AWS

[x] "Brexit Readiness" (working)
[x] "Icinga Service Status Overview (Chris B)" (working)

Integration

[x] "Brexit Readiness" (working)

Dashboards to delete

Production

[x] "App deployments" (no data)
[x] "Application 4XX and 5XX codes Copy" (duplicate)
[x] "Application 4XX and 5XX codes for whitehall and asset manager" (duplicate)
[x] "Application restarts due to memory usage" (duplicate)
[x] "Applications with no running processes (according to Graphite)" (no data)
[x] "Asset manager memory" (incomplete)
[x] "Asset Master virus scan speed (copy)" (no data)
[x] "Attempt at Error rate per month dashboard" (incomplete)
[x] "Content Pages Nav Test" (no data)
[x] "Content Performance Manager" (no data)
[x] "Content store consistency" (no data)
[x] "Deployment dashboard - design principles" (no data)
[x] "Deployment dashboard - link-checker-api2" (no data)
[x] "Deployment dashboard - whitehall prototype" (superseded)
[x] "Elasticsearch (Erin)" (incomplete)
[x] "Elasticsearch (Tijmen 3/9/2018)" (empty)
[x] "Email Alert API Metrics (Old)" (superseded)
[x] "Errors not reaching Sentry" (no data)
[x] "Finder requests & timing - Tijmen" (incomplete)
[x] "GOV.UK search indexes" (no data)
[x] "Government Frontend Duration" (no data)
[x] "Historic (Errbit) Errors" (no data)
[x] "How many errors did we serve during an incident?" (incomplete)
[x] "Invalid Problem Reports" (no data)
[x] "Link checker api and Redis" (incomplete)
[x] "Local links manager webhook timing" (no data)
[x] "Logit" (duplicate)
[x] "Memcached usage on finders" (no data)
[x] "Memory dashboard" (no data)
[x] "New Navigation" (broken)
[x] "Percentage of in scope content tagged to the Topic Taxonomy (Table)" (incomplete)
[x] "Potsgres Primary free disk space" (incomplete)
[x] "Prototype dashboard - Whitehall" (no data)
[x] "Publishing delay report" (no data)
[x] "Publishing delays - work in progress" (no data)
[x] "Rails Application Request Durations by Controller" (duplicate, no data)
[x] "Response times" (no data)
[x] "Rummager (temp PGoddard)" (incomplete)
[x] "Scheduled publishing Copy" (duplicate)
[x] "Search incident investigation 2017-10-31" (no data)
[x] "Signon" (incomplete)
[x] "TEST Platform Health: Q3 (19-20) Email alert API errors" (incomplete)
[x] "Tagging coverage" (no data)
[x] "Tagging interfaces" (no data)
[x] "Test (Jonathan)" (incomplete)
[x] "Whitehall (JN copy" (duplicate)
[x] "aws origin health" (broken)
[x] "origin 5xx splits" (no data)
[x] "temp - licensify errors" (no data)
[x] "whitehall" (duplicate)
[x] "Whitehall 5xx analysis" (duplicate)

Production AWS

[x] "Application dashboard - finder-frontend-temp" (broken)
[x] "Content Store Test" (incomplete)
[x] "Feedback honeypot effectiveness (#re-autom8 team)" (incomplete)
[x] "GDS API Adapters logged requests" (duplicate)
[x] "MongoDB (Chris B - For AWS)"
[x] "Puppet (Chris B)" (working)
[x] "Rails Application Request Durations by Controller" (duplicate, no data)
[x] "Sidekiq throughput (per machine)" (no data)
[x] "System file handles - Chris Baines" (incomplete)

Staging

[x] "Application restarts due to memory usage" (incomplete, duplicate)
[x] "Asset manager memory" (incomplete)
[x] "Content Store" (no data)
[x] "Deployment dashboard - design-principles" (no data)
[x] "Deployment dashboard - whitehall Joe Lanman" (incomplete)
[x] "Elasticsearch activity" (no data)
[x] "Info frontend" (broken)
[x] "Local links manager webhook timing" (no data)
[x] "Memcached usage on finders" (no data)
[x] "Memory Leak" (incomplete)
[x] "New dashboard" (incomplete)
[x] "PostgreSQL" (no data)
[x] "Prototype dashboard - Whitehall" (incomplete)
[x] "Publishing API Metrics Copy" (duplicate)
[x] "Rails Application Request Durations by Controller" (duplicate)
[x] "Rummager request performance" (no data)
[x] "Sidekiq Copy" (no data)
[x] "Sidekiq pub-api prcs/threads" (no data)
[x] "Smart answer locale stuff" (no data)
[x] "Tagging interface swarm" (no data)
[x] "Topic Taxonomy" (duplicate)
[x] "Whitehall 5xx analysis" (duplicate)
[x] "Whitehall Publishing Load" (duplicate)
[x] "email-alert-service" (incomplete)

Staging AWS

[x] "MongoDB (Chris B - For AWS)" (duplicate)
[x] "New dashboard" (no data)
[x] "Search requests and errors" (no data)
[x] "System file handles - Bevan" (incomplete)

Integration

[x] "Brexit Readiness - old" (duplicate)
[x] "System file handles - Peter" (incomplete)

CI

[x] "A horse, a horse! My kingdom for a horse!" (incomplete)
[x] "Assets" (no data)
[x] "CI GitHub Rate Limiting" (no data)
[x] "CI Machines" (duplicate of Machine)
[x] "CI performance" (no data)
[x] "Content Store Traffic" (duplicate)
[x] "Content validation content store" (no data)
[x] "Elasticsearch activity" (no data)
[x] "Email Alert API - 21/11/2017 WIP" (no data)
[x] "Email Alert API Metrics Test" (no data)
[x] "Email Alert API Metrics Testing" (no data)
[x] "Graphite Carbon Metrics" (incomplete)
[x] "Link Checker API and Redis" (no data)
[x] "Link checker controller duration" (no data)
[x] "Local links manager webhook timing" (no data)
[x] "Publishing API cache" (no data)
[x] "RabbitMQ" (no data)
[x] "Sidekiq (Older versions of govuk_sidekiq)" (no data)
[x] "Tagging Dashboard" (incomplete)
[x] "Tagging interface swarm" (no data)
[x] "Templating test" (broken)
[x] "Test dashboard please ignore" (empty)
[x] "Topic Taxonomy" (no data)
[x] "Topic Taxonomy Content Coverage" (no data)
[x] "Topic Taxonomy Content Distribution" (no data)
[x] "Uptime Metrics Experiments” (no data)

issyl0 commented 4 years ago

Given that all of the CI dashboards have no data and I'm not sure many people realise that CI even has a Grafana, let's delete them rather than fixing them.

issyl0 commented 4 years ago

We also don't need "Historic (Errbit) Errors" (no data), because we haven't run Errbit for years.

benthorner commented 4 years ago

I've now removed all the "Dashboards to delete" and exported/uploaded all the "Dashboards to keep" here, with a view to deleting them shortly unless they are urgently required to exist.

Assets.txt AWS second line.txt Brexit Readiness.txt Fastly CDN.txt GDS API Adapters errors (Chris B).txt Icinga Service Status Overview (Chris B).txt Search requests and errors.txt

sihugh commented 4 years ago

I'm going to revamp Search requests and errors at some point. It can die for now 👍

benthorner commented 4 years ago

The new alert is now deployed and all is green. We're done here 🎉

alphagov / govuk-puppet

Grafana dashboards aren't in version control #9937

Dashboards to keep

Production

Production AWS

Staging

Staging AWS

Integration

Dashboards to delete

Production

Production AWS

Staging

Staging AWS

Integration

CI