A Dashboard for Engineers

adamlofting commented 9 years ago

Things can go wrong with our products without us knowing, and it's hard to see at a glance where efforts are best spent optimizing our engineering output across our many sites and repositories.

We want to give engineers clear visibility into the metrics they have agency over, in order to improve the experience for end-users of our products, and our contributors, and the engineers too.

The metrics discussed here are those specific to the engineering domain (unlike those which are also impacted by design, UX, marketing, product, etc).

Audience

MoFo engineers, and engineering contributors.

Success

Engineers can see at a glance:

Pages with slow front-end performance (page load time)
Routes with slow back-end performance (server response time)
Pages with too many http requests to a single domain
Unanswered volunteer input in Github issues
PRs that have had no response at all
PRs that have had no activity for more than X days
Number of open bugs (total)
Number of bugs needing triage
Number of bugs by triage status
Number of bugs assigned to expired Milestones
Repos with code coverage below X %
Current hosting costs (AWS + Heroku + Services)
TBC: There's a moco tool that scans a site for things like CSP, X-frame-options etc as a base level of sec requirements. Perhaps we could surface results from that? I can't remember the name right now.
TBC: Something like Gemnasium alerts for old dependencies (+ security alerts)
TBC: Something like a Codeclimate score?

What else would the engineers like?

This is not a data reporting exercise (e.g. avg. response time to PRs over time) but instead an actionable list driven by data.

Vision

By separately surfacing the problem areas in a single dashboard, we have a single list of actions to take. This will hopefully be more effective at getting the problems solved than having to look through many larger reports to identify issues.

Our work is distributed across many repos and projects, and the tooling for this dashboard may require a list of 'repos we care about' to run the analysis against.

Without wanting to make this task bigger than it needs to be. This feels like a tool that could be useful to many people with an open source project and many repos to maintain.

Measurement

Adoption of the dashboard(s) by engineering team
Regular referencing of the dashboard(s) in engineering (and/or Plan) meetings
Improvement in baseline results for the things we surface, and maintaining this improvement over time
Roles
Phase: Scoping
Driver: @adamlofting
Decision Maker: @simonwex
Design Lead:
Development Lead:
Quality Verifier:

adamlofting commented 9 years ago

cc/ @simonwex (I'm going to file a separate ticket for the DevOps piece)

thisandagain commented 9 years ago

Excellent. Some of these items are particularly helpful for user support and prioritizing triage as well. The only thing I would add is error counts / rates (HTTP errors and exceptions from both API and client-facing services).

simonwex commented 9 years ago

@adamlofting I believe the sec tool you're talking about is Minion. It's in the middle of a complete overhaul, we should check in with Yvan to see where that's at.

jdotpz commented 9 years ago

I did a quick catchup with @adamlofting Much of the existing dashboarding is in New Relic, so that will need some new options. For error reporting, Loggins is a possibility. For performance, well, not really too sure you're going to get the same views those dashboards currently give, at least not without writing Newer Relic. :) For security, the MoCo security team itself told us to look at Whitehat Security. I've got a conversation going with them that I'm happy to transition over to whomever will handle that.

simonwex commented 9 years ago

/cc @jbuck

ScottDowne commented 9 years ago

@adamlofting if you need this, I started a list of mozilla org repos: https://gist.github.com/ScottDowne/88b8728b0757b8d574d3

It's sorted by most recent activity at the time I grabbed them.

This doesn't include MozillaFoundation org repos because that's not a silly amount of repos and is fair to just include them all for now.

I do feel like we should have our own org for some of these if possible, makes it easier to do things like find a list of things we care about. It also means we can ask github for a list of repos under "mozilla-foundation-software" org and not get thousands of repos back.

adamlofting commented 9 years ago

Adding a milestone so this can surface in the planning meeting at somepoint. I suspect resourcing wise it will be lower priority for now.

adamlofting commented 9 years ago

No developer resource assigned to this for now, so moving it out of this heartbeat.

adamlofting commented 9 years ago

A couple of contributors have reached out to me, and are interested in working on metrics work so we're going to start with this engineering dashboard project.

I've setup a new repo to use as a workspace here: https://github.com/adamlofting/dashboard-zero

And we'll start by focusing on the Github data as that feels like the lowest hanging fruit.

MozillaFoundation / plan