Graph Request: PR Workload

[jberkus]

Here's another graph I would find useful: PR workload per SIG. I think this would probably work best as a heatmap, with SIGs on the y-axis and time periods (months or quarters) on the x-axis. I would like one both for absolute and relative workloads.

The scalar for "absolute workload" would be defined as:

(# of new PRs) * (size of new PRs)

And for "relative workload" as:

((# of new PRs) * (size of new PRs))/(# of unique reviewers in the SIG that period)

To multiply "size" of PR, we would index the size labels as follows:

• XS = 0.25
• S = 0.5
• M or not specified = 1.0
• L = 2.0
• XL = 4.0
• XXL = 8.0

Potential blockers: may PRs don't get a size and SIG label in the month they are filed. However, the main value of this chart is seeing how loaded down the SIGs are. Ideas?

[dankohn]

Cool idea.

[lukaszgryglicki]

Great, I just need a priority order of all issues reported on cncf/devstats. Probably will need some additional Go code to handle data for heatmaps - but that seems to be a fun.

[jberkus]

Will discuss priorities at next Contribex meeting.

[jberkus]

Also, question: does it help for me to supply PostgreSQL queries for my graph requests?

[dankohn]

I believe the actual queries will be to InfluxDB, but a Postgres query would be great to specify exactly what you're looking for.

[lukaszgryglicki]

I don't yet know what Grafana expects for heat map. The first query will be on PSQL to get data and to save its result to Influx. The second query will be from Influx to get data on Grafana.

The first query runs once per hour so can be very very complex. The Influx query should only request "linear" data from Influx (data saved by Postgres). It is always something like select * from series_name where time_filter or something very similar. This must be hyper fast and very simple.

That's the architecture.

And yes please send PSQL query if you have one, it will be tweaked for sure, but will be a very good starter.

[lukaszgryglicki]

For example, let's take "Activity repo groups" metric, it is defined here:

• https://github.com/cncf/devstats/blob/master/metrics/kubernetes/metrics.yaml#L5 It runs this query (every hour):
• https://github.com/cncf/devstats/blob/master/metrics/kubernetes/activity_repo_groups.sql It uses some templating variables inside {{ and }}. This saves to repo_group_activity_* series (multivalue in this case) Finally, Grafana takes values from Influx using this:
• https://github.com/cncf/devstats/blob/master/grafana/dashboards/kubernetes/activity_repo_groups.json#L115

[lukaszgryglicki]

I'm starting to research what I need for heatmap (what Grafanba requires) and then checking if I can do it with current implementation - or if I need to add some Go code...

[lukaszgryglicki]

I'm not sure how to use heatmaps here... http://docs.grafana.org/features/panels/heatmap/

• X-axis would be time series (months, weeks etc) - OK
• Y-axis - you want me to use SIGs as Y-axis - but from Grafana docs it seems like Y-axis must be numeric
• Z-axis (color/heat) - PR size - also OK.

Hmmm, I'm kinda blocked, ideas?

[jberkus]

Hmmm. Yeah, looks like Grafana treats heatmaps exclusively as an extension of histograms, and as such requires all axes to be numeric (or time), rather than quanta. Lemme poke around and see if there's some way around this, because there's other places I'm going to want heatmaps for non-scalars.

[jberkus]

BTW, you could obviously do this as a stacked graph, but that's harder to visually process. A Heatmap makes it easy to see if a SIG is chronically overloaded, or if a specific time period is heavy load across all Sigs, in a way that's hard to see on a stacked graph.

[jberkus]

So, I've been looking over heatmaps and heatmap plugins for Graphana, and the kind of heatmap we need isn't available. We'd have to hack something ourselves to get this.

However, the Table Panel looks like it might be adaptable to what we need for this, in a rough way: https://grafana.com/plugins/table

We might have to turn the chart 90deg (that is, have time periods down the side and sigs across the top) but that would be OK.

[lukaszgryglicki]

I will check the table, actually, we have few tabular dashboards that use time on the Y-axis and multiple values on the X-axis. For example here: https://k8s.devstats.cncf.io/d/000000017/opened-to-merged?orgId=1

[lukaszgryglicki]

I can make it similar to this: https://k8s.devstats.cncf.io/d/000000073/project-statistics?orgId=1

• You will have SIGs in rows (each row will be a single SIG).
• You will have absolute and relative workload on the columns, so columns will be: SIG, Relative workload, Absolute workload, # of reviewers.

Where should I get the number of reviewers in a given SIG? There will be no repo group drop down for this dashboard. there will be range drop-down - to allow selecting: last year, decade, month, week, inter release ranges etc (just like in the link provided).

Will it work for you?

[lukaszgryglicki]

I'm starting work on this. @jberkus I need to know how to calculate number of reviewers per SIG. Because I don't know this yet, I'll now implement tabular view as described - but with only SIG and Absolute PR workload values.

[lukaszgryglicki]

Initial SQL will be something like this: https://github.com/cncf/devstats/commit/b14bd1d8cd3c0529365bf8e45bda1dc5399b6e0d I'll create both tabular and chart dashboards. I'll add relative workload values once I know how to calculate the number of reviewers per SIG.

[lukaszgryglicki]

Actually, I need a new special tabular type data processing for this, so it will take a little time anyway...

[lukaszgryglicki]

I'm working on this but in the meantime, it would be good to get some feedback:

• If we're going to show a chart (like many dashboards do). Which PRs should we count at the given point in time? Those opened before this point and not yet closed? Does all PRs get closed after they're finished (merged or discarded)? We need to define this very clearly: we have many dates here: PR creation date, PR close date, PR merge date, Issue addition date...
• If we're going to show this as a tabular view, then we have date ranges, like last day, week, month ..., between v1.X and v1.Y - I also need a clear definition which PRs should I take into account (see dates described above)
• And once again - to get "relative" number - I need a way to define number of reviewers per SIG.

I will only make some example demo today - far from being final.

[lukaszgryglicki]

Very very initial example of tabular data: https://k8s.cncftest.io/d/QQN85o3zk/pr-workload-table?orgId=1

[lukaszgryglicki]

Ad this is a chart-like dashboard preview: https://k8s.cncftest.io/d/hnphTo3kk/pr-workload?orgId=1

[dankohn]

The % version of PR workload looks like modern art.

[lukaszgryglicki]

To continue work on this task I need more info from @jberkus

[lukaszgryglicki]

Added current dashboards on the prod server, they only have "absolute workload per sig":

• chart.
• table.

[jberkus]

The table is good, but I'd really love to also have the values over time, which is why I was suggesting a table plugin. I'll create a mockup of what I'm talking about once I'm back in the office, and if I can also figure out the table plugin I will.

[jberkus]

Oh, also, are these PRs by creation date? or concurrent load of all open PRs?

[lukaszgryglicki]

It's by PR's creation date. I can change to the number of open PRs at given point of time (this makes sense for the chart version, for table version where we're choosing date range - PRs created in this date range will work better IMHO). If I change chart version to use "point in time state of open PRs" then Period drop-down no longer makes sense, and I will remove it and chart will show daily values - which means the number of open PRs at any given day in the past. I'm implementing https://github.com/cncf/devstats/issues/77 that way. There are table and chart version. I'm waiting for mockup, I'll update as needed.

[lukaszgryglicki]

BTW: table allows to select values over "quick ranges" - You can select last day, 10 days, week, month, quarter, year, decade, and values between every k8s release and next release. I can try to create another table view where rows will be SIG and columns can be Absolute PR workload date1-date2, then Absolute PR workload date2-date3, ... and so on. But I would need to know which dates are needed, this should be configurable somewhere in the metric definition. The next thing is "Relative workload", to calculate this I need to know the number of reviewers per SIG (asked few times). I don't know how to calculate this.

[jberkus]

Yes, for the chart I'd love to have PRs open on that date instead of PRs created on that date.

The problem with guaguing by PR created date is that it tends to obscure PRs which have been open for a long time, and those are often the most work-generating PRs. Some stay open for months. So, a different option for the table would be a date picker which simply allows you to pick a specific date (but not a range). Then it would make sense to do it as PRs concurrently open on that date. Even better would be a picker which does allow a date range, but then shows high, low, and average counts for concurrently open PRs on each day of that range. Gauge that against the amount of work it would be, though; 80% of the time, we're just going to look at the current day.

Reviewers per sig should be calculated by using the existing approver stats, per the approvers chart, except with SIGs added. That is, take the number of people who did approvals on that SIG's issues for the same time period (yes, I realize some approvers will be counted twice because of multi-SIG PRs, but that's not a problem for this table). If we going for picking a single date and not a range, it would be approvers during the week leading up to that date.

[lukaszgryglicki]

Seems complex at first glance, but doable, I'll take a deeper look next week.

[lukaszgryglicki]

Back on this. I'll work on the "chart" approach, but change from "PRs opened during the period range" to "PRs opened at this date", so the Period drop-down will be gone (it will just conatin daily values). I'll add relative workload based on number of reviewers in a given SIG as suggested by @jberkus I'll count SIG reviewers by:

• counting people who added "approve" and/or "lgtm" label on this SIG's issues during the week before "date to".
• counting people who added "/approve" or "/lgtm" text on this SIG's issues during the week before "date to".

[lukaszgryglicki]

I think that counting reviewers from last week gives too chaotic results (changes often). I would count reviewers from at least on month? Seems OK? I'll change to month ayway, and see if data looks better.

[lukaszgryglicki]

This is the new chart. It contains daily value, each day means "PR workload" state at this day (not PRs opened on this day etc). There are 3 charts of absolute PR workload. 3 charts of relative workload. Chart of the number of SIG PRs opened. Chart of the number of SIG reviewers.

https://k8s.devstats.cncf.io/d/hnphTo3kk/pr-workload?orgId=1

For the table version (https://k8s.devstats.cncf.io/d/QQN85o3zk/pr-workload-table?orgId=1) I think I should count reviewers in the selected date range (like v1.8-v1.9 or last month) and display Opened PRs state at the end of that range.

Will start working on this today. I also need to update test coverage, current changes broke it,a nd I wanted to show new version before end of work, so I can get feedback, and if approved I'll update coverage, if not - then I'll do changes to dashboards saving multiple test coverage fix work time.

[lukaszgryglicki]

Tabular version now also have:

• Relative PR workload
• Number of reviewers.

Table version counts reviewers in a date period selected from the drop down. It displays PRs state at the end of that period.

So for example "last month":

• Counts reviewers from the last month
• Displays current PRs state

"v1.9 - now":

• Counts reviewers since v1.9 release
• Displays current PRs state

"v1.8 - v.19"

• Counts reviewers since v1.8 release to v1.9 release.
• Displays PRs state at v1.9 release time.

https://k8s.devstats.cncf.io/d/QQN85o3zk/pr-workload-table?orgId=1

Awaiting feedback for both "PR workload" and "PR workload table".

[jberkus]

Wow, that's awesome! I mean, really nice.

Closing this, that does everything I was looking for.

[lukaszgryglicki]

OK, so I'll update test coverage for those dashboards.