Create Metrics to measure developer experience (& evaluate our impact)

[galargh]

Merge time,

Pipeline run time,

etc.

We don’t want to make these a goal by itself (”let’s improve the PR merge time metric by not reading this code!”), but identify outliers and eventual issues.

Things we could do:

Start gathering github metrics about PRs, Issues, and releases

[galargh]

Maybe some metrics to try to measure developer pain? (number of minutes CI running/number of PRs, number of PR CI failures, number of "old code" lines removed in a period of time, and so on)

[laurentsenta]

This was discussed during a triage session, CI pipeline execution time might be a good start here, see: https://github.com/libp2p/go-libp2p/pull/1639#pullrequestreview-1031484125

[galargh]

Following up on the discussion, I think these would be useful:

• total wall clock time for workflow (might be the easiest to get)
• number of runners in use over time
• sum of time spent in queue (maybe we could get it per job and have a way to group by workflow - similar for execution time)

[laurentsenta]

@galargh I can't find the note from our IPDX session, when you're around could you share it?

What I recall:

• We want to measure the wall clock time from a workflow being scheduled to be completed.
• With this metric we can deduce
  • The time a maintainer has to wait before being able to merge,
  • Stragglers, if the metric suddenly goes up for a repo: a job might be blocking the pipeline,
  • Other nice things (total time per PRs, etc).

[galargh]

That discussion happened after we put our laptops away so there's no written record. But you're right, that's the gist of it. The wall clock time per workflow, consider using a new org for running the metric gatherer to not interfere with the real workflows (or maybe even go off Actions completely), send it to Grafana, describe everything so that it's easy to add new metrics.