To deprecate or Not: what information is useful for a dashboard

[victorb]

Raising this issue as ipfs/project-repos has not got a lot of attention and not a lot of usage so far. We should decide if we want to improve it enough to become useful, or deprecate it instead.

First question to solve is what information we want to get from a project-wide dashboard. Currently, the website is showing the following columns:

• TravisCI status (will soon be deprecated)
• CircleCI status (will soon be deprecated)
• That readme exists
• That readme has more than 500 characters
• LICENSE exists
• PATENTS exists
• README has the following sections:
  • ToC
  • Install
  • Usage
  • Contribute
  • License
  • TODO
  • Banners:
  • Travis
  • CircleCI
  • Made By
  • Project
  • IRC
• Number of Github stars
• Number of Github issues

I argue that most of this columns are not needed for daily inspection. Probably the ones that are useful right now is the current CI status, but that's coming from my need of investigating the CI health in general.

Is there any columns from the exist list that are actually useful to see day-to-day?

ipfs/js-ipfs has a "packages" list (https://github.com/ipfs/js-ipfs#packages) that shows the following:

• Version
• State of dependencies (up-to-date or outdated)
• Different status on CIs
• Current Code Coverage

Integrating those columns into project-repos would be useful.

Are there other metrics/checks that are useful to see on a regular basis?

[warpfork]

When I'm checking in on a project, I've often valued getting quick impressions of how much work is "in flight", because that affects how I'll plan and who I'll communicate with when considering larger refactors or other changes likely to generate conflicts (e.g. upgrading dependencies). Thus, some quick scores for "how many PRs are open? How many branches? How far diverged?" would be really awesome.

This is roughly the same as what git branch -vv can also tell you, but having it tersely available for a whole array of repos is nice.

A stateful service could also provide updates on when these things last changed. (Or, possibly this could just be done with git author time, or similar indicator.) This is useful in assessing overall health questions like "is there a lot of work 'in progress'?" and also "is there a lot of work that was in progress but is unmerged and now getting stale?"

A particularly advanced form of the "how should I plan for changes likely to make conflicts" would be examining actual diffstats and possibly even doing trial rebases/merges to see if they're going to generate conflicts. Knowing when a branch is high-impact is really helpful. This (especially trial rebases) takes a fair amount more compute effort to check, but as a proof of possibility Github also does some of this work internally (the buttons for "merge" on PRs do roughly this). Getting that info in a dashboard overview of branches (especially if we can then do summary counts e.g. "how many branches weigh over $X lines of diff", "how many branches conflict"), even though it's some work to implement the who "try the rebase" ourselves, seems like it would be super useful.

Some of these things are relatively high-effort, but, they're also all project/repo/language agnostic (as long as "repo" is implemented by "git"), so IMHO the payoff is there, and it could make sense to make these stats uniformly available for every project in a dashboard.

[warpfork]

For what it's worth, I like the README, PATENTS, and LICENSE checks. I agree with questioning whether they deserve as much screen real-estate as they currently get, but I like them being somewhere. An aggregate "matches project standards" would perhaps be sufficient?

It's also interesting to note those README, etc checks seem to belong to a category of linters that can be run pretty low frequency, and are correct to memoize per commit hash (as opposed to things like github star count, or branch freshness, which cannot be memoized in such an easily-standardized way). So maybe these memoizable-per-commit category of checks could belong to some kind of very basic plugin system for content-based linters.

I had to think for a few minutes on what an argument is for having those checks separate from CI in the first place, since that's the usual home for per-commit checks. I guess it's because as a user story, "maybe there's things we want to check across projects"... and if nothing else, those checks would include checking for the checkers in the CI conf, neh? So really it's correct to memoize not on the subject repo commit hash alone, but on the tuple {subjectRepoHash,dashboardRulesVersionHash} -- and that's why it makes sense not to have those linters configured in the project's individual CI.

[RichardLitt]

I argue that most of this columns are not needed for daily inspection. Probably the ones that are useful right now is the current CI status, but that's coming from my need of investigating the CI health in general.

Is there any columns from the exist list that are actually useful to see day-to- day?

As far as I know, daily inspection wasn't the goal of project-dashboards. It was for routine inspection on the level of documentation. If you want daily inspection, this tool should be forked and most of the fields removed as irrelevant.

[daviddias]

@ipfs/community-wg putting the Project project-repos on your radar.