New Metric: Change Request Reviews

[vinodkahuja]

This issue is created to collect comments about the rolling release of Change Request Reviews metric

This metric can be found here: https://docs.google.com/document/d/1AJ8NLoUQx84mTAFiTQpTj8lcOWF79_KV-xX7I9R21Sc/edit

See all release candidates: https://chaoss.community/metrics/

CHAOSS Metric Quality Checklist

This checklist is used for new and updated metrics to ensure we follow CHAOSS quality standards and processes. Below checklist items don’t have to be completed all at once: create the metric release candidate issue first and then start working on the checklist.

Process

• [x] Create the “review issue” in the authoring WG’s repo for comments during review period and paste this template in
• [x] Create pull request to edit or add metric to WG’s repo (after checking Content Quality and Technical Requirements below)
• [x] Add the new metric or metric edit to release notes issue in working group repo
• [x] Update the Metrics Spreadsheet
• [ ] Create issue in CHAOSS/Translations repository to kick-off translation to other languages (please use the the translation issue template)
• [ ] "Metric Candidate Release" label added to the metric release candidate issue.
• [ ] Metric was added to website

When above steps are completed:

• [ ] Announce new/updated metric on mailing list, newsletter, community Zoom call, and Twitter. This can be coordinated with the community manager.

Content Quality

• [x] Required headings are filled in, including Questions.
• [x] Description provides context to metric
• [x] Objectives list sample uses for the metric and desired outcomes
• [x] If any, DEI uses of the metric are included in Objectives
• [x] Optional headings that have no content were removed
• [x] Contributors section lists those contributors that want to be named
• [x] The name of the metric is the same in (1) metric heading, (2) metric file name, (3) focus area, (4) metrics spreadsheet, (5) “review issue”, (6) translation issue, and (7) website

Technical Requirements

• [x] Message in the metric markdown file that the metric will be part of the next regular release is at top of page and the links are correct (this is in the metric template)
• [x] Metric file name is the full metric name and only contains lower case letters and hyphens (“-”) for spaces
• [x] Images are included using markdown and relative links (as described in the metrics template)
• [x] Images have at least one empty line above and below them
• [x] Ensure images are placed in image folder and followed naming convention
• [x] If new focus area is created, ensure focus area is added to wg repo readme and focus area folder readme
• [x] Within the focus area, add the metric in the table and provide the link to the metric and metric question
• [x] Ensure tables within metric are converted as image and placed in the image folder (both original MD and screenshotted PNG format) and follow the naming convention
• [x] No HTML code in the metrics markdown file

[rpaik]

A few suggestions:

• In the first sentence under Descriptions, can we change "developers" to "community members". For things like documentation, anyone (incl. non-developers) provides reviews/feedback.
• Can we change the 2nd sentence under objectives to "Change Request Reviews can help inform the quality of the software and the efficiency of development."? (Software includes not only code, but also documentation, examples, tutorials, etc.)

[foundjem]

@rpaik I see what you mean. Indeed, I think “contributor” is widely used in place of “developer” because the former encompasses all those making any form of changes in the community repo.

Signed-off-by: foundjem@ieee.org

[geekygirldawn]

In the implementation section - should the text be bold? And I think maybe the link to the data ethics policy isn't right?

Can we add an image to show an example visualization?

[sgoggins]

@rpaik : We addressed the points you made. @geekygirldawn : We fixed the bolding. Right now we do not have a visualization, but we will aim to build one before the end of the review period.

[bhack]

Just few additional point:

• I think it could be interesting to collect also the review diversity, like we have in committers metrics (by domain/org).
• it could be nice to understand who is going to merge the CR (or to approve it as in large projects generally this is merged by a bot)
• It could be nice to have also an review requests/reviewers ratio. Sometimes many reviewers are requested but just a few or a single one review (especially on github with CODEOWNERS.md review auto-request)
• New reviewers are requested over time specially on large repositories cause reviewer unavailability. I suppose we could extract some numbers here.

[germonprez]

@sgoggins, what is the full status of these reviews? It looks like the last comment from @bhack is not addressed.

Also, do you have a visualization that was going to be included as part of the review process?

[klumb]

@bhack, those are all good ideas but I believe they describe different metrics or models. The goal of this metric is to measure to what extent are change requests put through a formal review process using platform features? It can inform other metrics or models but it is basically a count of reviews in a change request.

We do need to add a DEI objective though. That is missing. Perhaps per @bhack comment, the DEI objective should be: "Exploration of Change Requests Reviews along with demographics of participants may highlight issues of DEI in a projects formal review process"

[klumb]

Also, I don't think the image needs to be a requirement for this metric to be released

[bhack]

Some of these could be routed to the DEI team. Others I suppose that could be new metrics as some pure metrics about the ratio of code gatekeepers/codeowners and submitted PR could be interesting also without an analysis of the DEI dimension.

[klumb]

I believe this metric is now ready for release.