search

ipfs-shipyard / ecosystem-dashboard

A dashboard for monitoring the ecosystem around an open source project.
https://ipfs.ecosystem-dashboard.com
27 stars 15 forks source link

Understanding contributor metrics #470

Open BigLep opened 2 years ago

BigLep commented 2 years ago

My end goal here is to have metrics for the IPFS project that give signal to whether we're adding "meaningful contributors" (rather than net-extractive contributors). As a result, I have a few questions:

Using https://ecosystem-research.herokuapp.com/home?org=ipfs as a starting place...

  1. What constitutes someone as a "contributor"? Is it if they made a "contribution" (meaning opened a PR or Issue per https://ecosystem-research.herokuapp.com/contributors?org=ipfs&range=7 )?
  2. Are there any other user actions/events that constitute as a contribution (e.g., commenting on an issue or PR)?
  3. Are contributions weighted in any way (e.g., a PR that gets merged is more significant than an issue that gets closed because it had no repo case)?
  4. What defines "new contributors" (https://ecosystem-research.herokuapp.com/contributors/new?org=ipfs&range=7 )? Is a "new contributor" a "contributor" who the system has never seen make "contribution" before to the filter area (e.g., org=ipfs in this case)?
  5. I know there's the option to filter "contributors" by organization or collab. Is it possible to filter by other criteria? For example, can I filter "contributors to our JS repos"?

I did a search through the readme and old issues, but didn't find these questions discussed. My apologies if these questions are covered other places. Please feel free to redirect me.

Cc @reidw since we were discussing some of this recently.

andrew commented 2 years ago

The word contributors has a couple of mixed meanings in different places across that app which comes from different sources of data for humans who have interacted with repos in some way that we can record, in two different tables:

  1. users listed on https://ecosystem-research.herokuapp.com/contributors?org=ipfs&range=7 are pulled from the Issues table, so only includes the users who opened the issue/pr originally, no comments, commits, reviews etc
  2. Event data is only currently used to guess if a user is likely a member of an org for collab purposes (PushEvent to a Chainsafe repo for example)
  3. there's no weighting, just counts for a given time range
  4. "New" means they have not opened an issue/pr before the start of the time range you're viewing, e.g. not opened an issue in that org before 7 days ago
  5. It could be possible to filter contributions by language, no-one has ever asked for that before so it wasn't implemented.

I'm not sure how aware of the PMF data that the ecosystem dashboard is also collecting but that does it in a much more indepth way using most event types to measure contributions (ignores ['WatchEvent', 'MemberEvent', 'PublicEvent'] event types as they aren't considered meaningful contributions.

There's definitely scope for redeveloping the contributor areas of the app to count and weight more types of contributions over time, it'll likely end up looking a lot like the PMF calculations, which that code is already written, just not used in the current browsable pages.

Happy to discuss further after I get back from holiday (17th october)

BigLep commented 2 years ago

Thanks as always @andrew for the help and explanations. Enjoy your holiday. We can connect more after you return. Thanks again!

BigLep commented 1 year ago

Thanks @andrew for the expalanations. These all make sense.

It sounds like the right next step is for me to make sure I'm up to speed on the PMF data. I believe the latest conversation on that topic is here: https://github.com/protocol/pldw/issues/166

Thanks again, Steve