create a tracking mechanism for qualitative SIG health

[bhs]

The OTel GC recently met to discuss priorities for 2022, and one is to improve both the OTel contributor experience and the OTel maintainer experience. As part of this, we want to do a more principled job of understanding the qualitative health of the many OTel SIGs.

My concrete proposal is to have SIG maintainers make qualitative assessments of their respective SIGs once per quarter using the following spreadsheet: OpenTelemetry SIG health tracker.

For those who don't feel like clicking on links, it looks like this (at this point, all of the letter grades are fake / just provided for illustration):

Comments very welcome from the entire OTel community!

cc @open-telemetry/technical-committee @open-telemetry/governance-committee

[jpkrohling]

I like the general idea, and I think we have great potential to expand this and transform it into recommendations and guidelines. For instance, what are the best practices when it comes to welcoming new contributors? Is it sufficient to have an up-to-date "contributing.md"? What's an "advancement opportunity"?

[sharrmander]

I also like the general idea - and to @jpkrohling's point I also think more guidance on how to grade is important to build confidence in the results. I recommend we provide a list of a few questions &/or guidance for each of the categories so the grading can be more objective.

Drawing out the example for being welcoming - something like:

• We keep our contributing.md file up to date
• We use the tags good first issue and/or help needed to highlight new contribution opportunities
• We have clear code examples that help new contributors come up to speed
• We explicitly thank new contributors upon their first engagement
• We acknowledge new PRs or issues with at least a comment within x amount of time

With a likert-type scale that looks similar to: N/A | Never | Rarely | Sometimes | Often | Always

The benefits of this is that the grading would be more consistent across groups, but the downside is that grading would take more time and we would need to come up with a list of categories and questions. I think the benefits outweigh the downsides though, because it can also help with setting context between communities about best practices too.

[bhs]

Hm... I also like the idea of providing more "OTel maintainer enablement", and I also like the specific suggestions (e.g., in @sharrmander's bullets above). That being said, I do not want the SIG health assessment itself to feel too prescriptive about potential solutions or tactics.

My other self-critique is that maintainers may simply not be the right people to assess the overall/holistic contributor experience for their respective SIGs. (And yet directly polling the contributors will be challenging, especially for those that did not get traction with OTel and churned out)

Curious what others think, of course!

[jpkrohling]

maintainers may simply not be the right people to assess the overall/holistic contributor experience for their respective SIGs

Agree, which is what made me realize that we'd all be rating ourselves "A" unless we have objective ways of performing the assessment.

[kenfinnigan]

Could maintainers from one group evaluate a separate group, rather than self evaluation?

[bhs]

@kenfinnigan I like this idea conceptually, but anticipate practical problems where maintainers from SIG A won't have enough context about the day-to-day in SIG B.

[bhs]

Okay, two revisions to the proposal:

1) Attempt to normalize the health assessments from the maintainers (without getting into prescriptive and/or elaborate rubrics)

Rather than just A/B/C/D, we could include some additional explanatory context for each letter grade, like this:

2) Attempt to promote accurate self-assessments by requiring maintainers to vet and discuss their self-assessments with their SIG contributors wherever intra-SIG public discussions typically take place (whether that be GH, Slack, or something else). As part of the grade, we would require the maintainers to post a link to where that self-assessment was posted and possibly discussed with their respective communities.

Also, I'm wondering if we want to include some sort of assessment axis for the number of active contributors... I expect that some SIGs are greatly challenged by the number of people who can actually make material contributions.

[kenfinnigan]

@kenfinnigan I like this idea conceptually, but anticipate practical problems where maintainers from SIG A won't have enough context about the day-to-day in SIG B.

Agree it could be a problem, as well as in terms of asking other groups to take the time to perform the evaluation.

On the other hand, it could be a yardstick as to whether the questions to be evaluated are easily evaluated from the outside