Measuring repository health

[chendaniely]

Our core lessons are in maintenance mode and undergo very few commits per week/year.

*code for plots can be found here: https://github.com/chendaniely/maintainers

Is there any good set of metrics that we can use to see if a lesson is being unmaintained? The number of commits was the easiest first pass of metrics to calculate and will probably not be useful in determining the "health" of a lesson since the lessons are not undergoing new lesson content.

• One idea would be to look at the time to issue and PR response when the issue is not posted by a maintainer.
• Another is to count the number of repository actions (issues created, PRs created, comments added)

It's also very likely that lessons will have 0 actions in a year due to the maintenance mode the lessons are in. Even with the instructor training checkout system, not every lesson will be used in the checkout process.

With the lessons naturally having a low activity, what would be something useful for current maintainers and administrators find useful to look at? I want to be careful we do not introduce any artificial metrics that end up being counter-productive for maintainers.

[ldko]

I agree that commits are not an accurate way to determine health, since longer established repos will often have less commits offered to it, may be less likely to be merged (since they are less likely to be obviously needed fixes such as typos and more likely about should we add new content/change the way we teach something), and can be impacted on count depending if someone merges with squashing or not etc. I agree that maintainers' responding to PRs and issues of external contributors is something that would be important to look at--if no one is responding at all--that is certainly an issue. Also if the content of the lesson no longer functions as expected--syntax changing in version upgrade etc. breaks the examples/exercises, that is not healthy.

[annajiat]

One single metric might not be able to capture different scenarios for all lessons.

Should we add a ping per year to ask

• if there is any latest version of required software and whether the setup instructions need to be updated or not?
• if there is any new research or industry practices that should be adapted

[bencomp]

Some time ago I was thinking about the idea of measuring the health of The Carpentries lessons too. In a podcast I had heard of CHAOSS, which provides tools for measuring activity in open source software. While interesting, it also feels very invasive.

As a maintainer (and potential contributor), I tend to look at how issues and pull requests are handled for a repository, as @ldko mentioned. How old are the open issues? Did they get labeled? Did anyone comment (and are maintainers involved)? Are there links to PRs or other issues? Is it clear what each issue is about, are they small and actionable or broad discussions? It is hard to convert these questions into objective measures or metrics, but I feel they are important.

A few days ago, I counted around 270 issues and PRs that were labeled "help needed" or "good first issue" in the Help wanted list, some of which were last active/updated over five years ago. I suggested a dashboard, not unlike the images in this issue, to view the activity of repositories over time. A dashboard may only help to find where additional (maintainer) resources may be needed, although it of course depends on the kind of issues to determine what kind of help is necessary.

So I agree that it is hard to get useful metrics for measuring health, even though there may be enough indicators.

[karenword]

tagging @klbarnes20 as this relates to assessment and metrics development

[bencomp]

Perhaps I should note that I am not trying to repeat the discussion that I started in Slack 🙂