Show GitHub metrics within pub.dev

[jayoung-lee]

When evaluating a package that needs regular maintenance, users visit GitHub to check the package’s recent activities and maintenance status. We can consider showing key GitHub metrics within pub.dev to save users’ trip to GitHub.

Key metrics that users mentioned in a recent user study were:

• Last commit date/content
• Number of open issues
• Number of open PRs

[YazeedAlKhalaf]

That's a really great and useful idea! I believe we could use the GitHub API to do this. npm already does this and even VSCode marketplace.

[elbeicktalat]

Very nice idea in this way developers safe time, also having this data will make easier choices between the similar package.

[stuartmorgan]

One complicating factor here is packages from monorepos (flutter/plugins and flutter/packages being prominent examples, but certainly not the only ones); repo-level metrics will be misleading for per-package evaluation.

[jayoung-lee]

@stuartmorgan That's a good point. I think we shouldn't display the metrics when there's no 1:1 match. Do you know how many packages are from monorepos?

[stuartmorgan]

I don't, although we could probably cobble together a script to make an educated guess based on crawling pub.dev links. The problem with not showing anything would be that it would likely be confusing to have a lot of the most popular packages (from a quick manual check, it looks like about 50% of Flutter Favorites, for instance) not have the quality metrics.

[leighajarett]

+1 This came up during research sessions for Flutter Vikings. Developers were also interested in seeing security vulnerabilities broken out as their own category if possible (perhaps through the use of Github tags), and # of tests.

We also talked to developers about being too intertwined with Github, but everyone we spoke with was happy to only use Github for hosting their packages as opposed to other git providers.

@stuartmorgan in the case of monorepos: could we also try to look only for commits in directories that match the package name, and issue tags that match as well (or contain the name so it would catch our p: )? I know there would be some errors here, but maybe we just make that clear to both package consumers and developers so they're aware

[stuartmorgan]

could we also try to look only for commits in directories that match the package name

We'd want to spot-check some of the other monorepos, but that could likely work. (There are some subtle questions to resolve first about what exactly we even want to count when it comes to federated plugins.)

issue tags that match as well

Maybe? I just checked the Plus Plugins repo, and they don't appear to be labeling consistently.

I actually have significant reservations about issue counts as a metric we actively promote though. In practice, issue counts are measures of five things:

1. popularity - Flutter has 11k open issues not because it's orders of magnitude less well-maintained than most packages, but because orders of magnitude more people are using it and filing issues. Obviously that's an extreme example, but the principle applies within the package ecosystem.
2. size/scope - Same argument here.
3. age - (maybe; see next item) Flutter has more than 2x the issues it had when I started working on it; that's not because it's become more poorly maintained, we just get more reports than we can fix (which is common, and not inherently problematic) so accumulation is a function of time.
4. issue handling policy - I'm aware of some packages that run a bot in their repo that closes every bug that hasn't been updated in several weeks. That doesn't make them better maintained than packages with a policy of leaving all unresolved bugs or feature requests open (like Flutter mostly does), but changes issue counts dramatically.
5. bug reports and feature requests - there's no reliable way to distinguish between them.

So of five things that have major influence on issue count, only one (really, only part of one) is really any kind of indication of package quality/maintenance quality. There's so much noise that at best I think the number is meaningless. At worst, we would incentivize behavior that I would argue is undesirable, like running bots that close almost everything (which results in things like 20 one-off closed issues about the same thing, rather than one issue that collects discussion, votes, etc.)

[sigurdm]

I agree it is hard to tell anything for certain from many of these numbers, and many of them can be gamed to some extent.

Still many people use the github link to estimate how well-maintained a package is. So we should be able to extract something useful if we're careful.

• Maybe rate of commits (how many commits per week during the last year) could be a useful indication of project liveliness. github has an api for getting recent commit activity,
• Github stars: indicating how many people are interested.

[stuartmorgan]

If we are willing to do not-trivial queries, an interesting metric could be something based on the percentage of recent issue that a project member has interacted with in some way (comment, label, etc.) That would give some indication of whether someone is at least looking at issues.