Can we measure the OWNERship change of ReleaseFilepath? Brainstorm metrics.

[andymeneely]

@rikal is driving this, but @dani5447 is familiar with this area.

These OWNERs files are an intriguing data source (http://www.chromium.org/developers/owners-files). These files are version-controlled, and regularly change over time. They have a hierarchical structure that implies some sort of team. They have per-file regular expression-like rules that can make our data collection rather... interesting.

My initial thoughts about ownership is that we should analyze the change in ownership (ownership churn?) for a given file. Changing ownership can be healthy, or perhaps unhealthy.

Maybe there's also a measure of how much ownership a given developer has? Perhaps dividing their attention too much?

You'll need to:

• Go over Shannon's initial investigation of OWNERS in her GoogleDocs
• Brainstorm some metrics that we might be able to collect using OWNERs files, and in particular the change in ownership over time.
• Describe how we might represent ownership in the database, and collect the data efficiently

[andymeneely]

Take a look at Chris Bird's paper on major/minor contributors (and some of the papers that reference it) for some inspiration. They don't have explicit data on ownership, but maybe a comparison would be good? Explicit vs. actual. Bureaucracy vs. activity.

[andymeneely]

From @bspates: What if we have an owner who is a minor or non-contributor who is LGTM'ing things?

[andymeneely]

From @Felivel: try out Git by a Bus for ways of measuring ownership. http://dev.hubspot.com/blog/bid/57694/Git-by-a-Bus

[dani5447]

I'm having some trouble finding papers by Chris Bird. Does anyone know of a good place to look for it?

[Felivel]

Did you try his personal web site? http://www.cabird.com/

[dani5447]

Ah, ok thank you! That didnt come up in my google search for some weird reason.

[dani5447]

Brainstorming Notes from 9/9 Meeting with @rikal + my own additional ideas:

• High expertise vs low expertise OWNERS - come up with a threshold
• Can we look at when a developer’s first commit is and trace approximately how long they’ve been on the project? - familiarity information. Then relate that with whether they’ve been an owner/when they were initially added as an owner.
• Can we determine what the optimum number of owners is, based on rietveld reviews that have been LGTM but introduced vulnerabilities, then look at how many owners were on that file path? -- during meeting we determined this might not be useful or practical.
• There are sometimes OWNERS files in a directory directly, but sometimes that is missing and there is an OWNERS file a level or more up that becomes defaulted. Can we determine having a lower level OWNERS file leads to more code familiarity and fewer vulnerabilities?
• There are probably some OWNERS that review code more often/give out more LGTMs. Can we determine who they are? Do they also contribute code to the project or have they become more managerial?
• Do there exist some OWNERS that are still on the file but have become inactive? How long have they been inactive?

During meeting we agreed these last two ideas might be some of the more interesting ones.

[andymeneely]

Good stuff! Here are some of my thoughts on those: "First commit" - definitely easy to get. Seeing a "Time To Owner" would be an interesting analysis. I can see that number being too short or too long.

For the lower-level OWNERS idea, we might able to measure something like "number of files a developer owns" vs. "number of OWNERS a file has". That would be less directory-heavy.

Inactive owners: yeah if we combine this work with major/minor contributors we can see if these align together. Or combine this idea with managerial or not. Peter Rigby's and Chris Bird's FSE paper last year touched upon this briefly and we can explore it further.

I'm wondering if we subdivide the system into teams and measure the teams by defect density and vulnerability density, then we might be able to evaluate whether or not teams are on the right track or not.

Good! Keep going. Hammer these out to metrics and some sort of analysis. Don't let what we currently have of a code base get in the way of a good idea.

[dani5447]

Some initial metrics concepts: - to be reviewed with @rikal during meeting

• percentage of OWNERS that have made at least one commit every two months
  • this would be to determine what percentage of OWNERS are "active" as contributors
• percentage of OWNERS that have commented on OR approved/denied a code review every two months
  • this would be the OWNERS who are active as reviewers/participating on reviews
• percentage of OWNERS listed in files who have not been active according to the two above bullets - inactive OWNERS
  • OR use concept of major/minor contributors?
• percentage of OWNERS who are active in both ways according to the first two bullets (actively make commits AND review code)
• average, min, and max "time to OWNER" - the time from the developer's first commit to when they were added to any OWNERS file
• num CVEs for filepaths with least "time to OWNER" / num CVEs for filepaths with most "time to OWNER" - will reveal an impact becoming an OWNER too quickly might have?
• average number of OWNERS active on a review - for each review, the number of OWNERS commenting, approving, or disapproving a review.
• num instances of one OWNER disapproving a review, and a different OWNER approving it any time afterwards
  • could show a potential disconnect/room for vulnerabilities?
• num instances of one OWNER approving a review, a bug/CVE being found and fixed resulting in a new review, and then another OWNER approving that review
  • same sort of concept as above, except now a bug has been introduced in the mix.
  • Does having a different OWNER review the bug/CVE-fix result in better or worse code? Can this be measured???
• average, max num files a developer owns - how many files does an OWNER have to be up to speed on at a time?
  • for max num files, compare to num CVEs on that filepath
• % of CVEs on filepaths that do not have an OWNERS file directly on the lower level (reviews are approved by OWNERS on an upper file level)

[andymeneely]

My meeting with @rikal:

• The easiest way to go about OWNERS is to measure them at each release. We currently have two, but we can probably go to analyzing up to about 10 more releases.
• But, only analyzing OWNERS at releases means the "Time To Owner" metric won't be very accurate since we only view snapshots. We can probably write a separate script that collects that based on Git commits to OWNERS files. That will need some more design work and planning before we start it.

I like these metrics for being added to release_filepath

• Major/Minor contributors for a given file
• % of OWNERs who were active (i.e. committed in the past X months, or reviewed in the past months)
• % of contributors who were NOT OWNERS
• Number of OWNERS per SLOC for a given file

For the number of files that the OWNER owns. take a look at this paper and consider the Entropy metric: http://macbeth.cs.ucdavis.edu/enteco.pdf

[dani5447]

From meeting with @rikal :

• Evaluate all four above metrics for both external and internal contributors
• ask @andymeneely - Should we consider dependencies as they relate to teams and filepaths? Is there a way to tell if a contributor external to a team is contributing due to cross-team dependencies?

[dani5447]

From meeting with @rikal :

• DB Table concepts:
  • team_file : key, file, membership (joins with filepath table)
  • team_directory : key, directory, membership
  • team_dev : developer, teamnum (joins with developer table)
• ^ we’re going to store team memberships for both files AND filepaths (filepaths to keep the structure, files for shorter quering)
• Still need to figure out how to parse OWNERS file using chromium-defined rules (perfile, setnoparent, etc)
• Actors to be considered with regard to OWNERS : contributor, OWNER (can be an contributor, reviewer), external contributor, major contributor, minor contributor, active OWNER, inactive OWNER, active contributor, inactive contributor
• metrics per team
  • % active OWNERs ,
  • % active contributors,
  • % external contributors = num not owners /total num contributors *100
• metrics per file/filepath

  • OWNERs/SLOC

  • contributors/SLOC (Are this metric and the one below useful?)

  • external contributors/SLOC (see above)

  • % external contributions =(commits from not owners /total commits from all contributors ) * 100
  • average churn by owners
  • % churn by external contributors - for a given file or filepath, get the commits from each minor contributor/external, sum churn of all = total % contribution by external contributors

[rikal]

Just expanding on the notes from Danielle, The metrics under consideration right now are focused on these areas:

Team Makeup

• % active OWNERS : (Active = contribute a commit or review one at least once, or we could also measure over a certain time interval, but need to define the ideal lenght)
• % active Contributors : Active contributors (OWNERS+ external contributors)
• % External Contributors : Those not specified as OWNERS

Ownership Burden or Load

• OWNERS/SLOC

• Contributor/SLOC

• External Contributor/SLOC ? (We were not sure if the latter 2 would be useful as Danielle mentioned)

Share of Contributions (in File or Directory)

• % External Contributions (Commits from external contributors not OWNERS)

Size of Contributions: Churn (File/ Directory)

• OWNER Churn Avg.
• % External Contributor Churn

Further thoughts are on Change in team makeup for OWNERS over time, focusing on deletions (loss of knowledge), additions (increased risk due to inexperience?)…

[andymeneely]

Latest metric list:

Team Makeup

• % active OWNERS : (Active = contribute a commit or review one at least once, or we could also measure over a certain time interval, but need to define the ideal lenght)
• % active Contributors : Active contributors (OWNERS+ external contributors)

Ownership Burden or Load

• OWNERS/SLOC

• Contributor/SLOC

• OWNERship entropy?

Share of Contributions (in File or Directory)

• % Commits from Owners

Size of Contributions: Churn (File/ Directory)

• OWNER Churn Avg.
• Non-owner Churn Avg.

Further thoughts are on Change in team makeup for OWNERS over time, focusing on deletions (loss of knowledge), additions (increased risk due to inexperience?)…

Metrics we want to use but haven't figured out yet:

• Time to Ownership: time from first review or first commit until when you got promoted OWNERs