search

ansible-community / stats-collections

RShiny app to display statistics for the Ansible Collections
GNU General Public License v3.0
1 stars 2 forks source link

Time to first comment metrics #6

Open GregSutcliffe opened 4 years ago

GregSutcliffe commented 4 years ago

This should probably be restricted to maintainers - i.e. time to first comment from a maintainer. However, that could be tricky in bot-controlled collections community.general

@gundalow is community.general the only repo that uses a bot to do merges in lieu of just having maintainers?

Failing that, it could instead be "time-to-first-comment" from a past committer...

GregSutcliffe commented 4 years ago

THis gets complex fast, the only way to identify a maintainer in a BOTMETA-controller repo is by parsing the botmeta file and comparing to the paths touched in the PR.

For simplicity, I'll start by looking at the "author_association" field in the GitHub api for comments. This can have the "maintainer" value, so we'll look for that. This won't work for bot repos like community.general as only the bot is an official maintainer, but we can revisit that.

GregSutcliffe commented 4 years ago

First version add in fa8c148e7d5f146224c8990061e0e8d9afcb1bae - for now this is just a comment from any non-bot person other than the author. Will refine further later.

gundalow commented 4 years ago

@gundalow is community.general the only repo that uses a bot to do merges in lieu of just having maintainers?

The bot will be running on other repos, such as AWS, Network & Windows

For collections using Zuul-CI (mainly network) all the merges are done by the Zuul bot when the gate label has been added AND CI is green. Example: https://github.com/ansible-collections/cisco.nxos/pull/146

This should probably be restricted to maintainers - i.e. time to first comment from a maintainer. However, that could be tricky in bot-controlled collections community.general

What are your thoughts around "Time to first comment by maintainer" vs "Time to first comment by some human"

GregSutcliffe commented 4 years ago

My gut feeling is that the only case where they'd be very different is when there's a lot of "+1 I have this issue" going on. Otherwise, I feel that most of the time, maintainers or keen contributors would be the first to reply.

Which makes a good point. If a maintainer is burning out, do we filter to just comments from maintainers (so we see the worsening time-to-comment) or include other humans (so that we see people are still getting replies)...

GregSutcliffe commented 4 years ago

Detecting maintainers is starting to look hard in a variety of contexts, but if we can do it, then I guess we'd like both?

gundalow commented 4 years ago

I'm OK with time to comment from first human (that isn't the creator) to start with.

For repos not using the Bot, maintainers are defined as those with write (or above) privileges, which I assume GitHub can give us.

GregSutcliffe commented 4 years ago

The API returns an "AuthorAssociation" field for each comment. https://docs.github.com/en/graphql/reference/enums#commentauthorassociation suggests that MEMBER or OWNER should tell us who can commit, normally. At least I think so - one has to be in the org to have rights granted, yes? But those rights might be "readonly"? Hmmmm.

GregSutcliffe commented 4 years ago

Ah, I think https://docs.github.com/en/graphql/reference/enums#repositorypermission will work. Will test later on.