Notes from LFLS

[germonprez]

Notes from LFLS -- Need to get these into the Indicators metrics in a systematic way.

• Metrics on your own employees (for example – contributing to projects)
• Risk mitigation
• Single points of failure – This is an ecosystem question
• Activity is not the only check
• Issues of Maturity (context)
• Style of Project (context)
• Quality of Open Source Ecosystem
• Data driven from metrics
• How are the H&S Metrics connected to value – Drive to Value
• Over time: How the metrics look now and they getting themselves into trouble.
• Number of YouTube video mentions
• Number of blog post mentions

Projects that aren’t actually abandoned but might be healthy

• Libraries.io

The ability to determine upstream and downstream dependencies

1) Project qua project
2) Are the people footing the bill happy with the h/s of the project – progressing 3) The external users are not necessarily a monolithic group

Users vs. builders

Comparative Measures vs. Absolute Measures – This is an interesting observation.

• How to compare between like projects.
• Figuring out which fork I should be using (this is sort of an example)

Classify projects – is there a way to automate this?

Activity is only part of the picture

to create: a universal set of health and sustainability metrics

Potential Categories (not sure how these line up with our current categories) Activity Maturity - If foundation has that info, then we can get it out of band. Viability (Bus Factor) Diversity

Interrelationship between health silos – Correlations

More Focused Metrics:

• Un-reopoened issues (or conversely reopened issues)
• Dependencies on Upstream Projects
• Number and velocity of releases (Major vs. minor releases).
• Code coverage
• Post release –> new bugs
• Stars for Public Relations – Velocity of stars

[GeorgLink]

I am thinking to wait until after LFOSLS to update everything at once.

[germonprez]

Ok.

[GeorgLink]

Note to myself:

• Aggregate of metrics from upstream projects.

The idea is that a package that uses a package that uses.... that uses a library that has not been maintained for 7 years .... might not be good despite that the first level is actively maintained.

[GeorgLink]

Email from Matt: Here is a repo very much worth checking out. https://github.com/brianwarner/facade

[GeorgLink]

Excerpt from notes 2nd day - Metrics:

• Distributed decision making – Kybernetes: Special Interest Groups
• Transparency measured through discussions (# of comments per commit)
• Roadmap (Kybernetes has a separate feature repository with the roadmap and issues)
• Community Meeting (Kybernetes demonstrates new features and gets updates from SIGs)
• F2F Community Gatherings
  • Getting people together resets contentious issues.
  • Allow for rivalries and disagreements to be resolved. Avoid long standing grudges that poison the community.
• Role definitions – foster taking responsibilities. – provide respectful ownership – refer to role owner even when you have a lot of experience and knowledge
• rewards / shout-outs / recognition. e.g. pull-request/change log mentions
• Retrospective (Lessons-learned, future improvements) meetings after each release
• Post mortem – meeting when something goes wrong
• release notes are complete and informative (maybe look at number of pull requests and number of items on release notes? – assumption is that each pull request is the unit of change)
• The node.js keynote showed the onion model (TC, Committers, Contributors, Users) and declared that sustainability is an issue, when the lines get too far away. Rough rule of thumb: 10x from each layer down.
• Path do leadership came up several times. One strategy is to focus on the lease likely contributors and making it easy for them to contribute. Then the high level contributors will also have an easier time to contribute (a side effect of fostering diversity) [find recording of node.js talk again]
• Downloads
• Rivalry or unity in community (maybe use sentiment analysis?)
• Diverse affiliation (Openstack requires no more than 50% in a project to be from one company; a project that violates this does not receive promotions, resources, or the same treatment at events)
• Use of acronyms, specialized language – can be a barrier to outsiders to join. – Language needs to be normalized between different companies that want to work together in an open source community (example was given from an Erickson view in the Telecommunications sector – telcospeak)
• # of jobpostings that reference a project

[GeorgLink]

Excerpt from 3rd day note: metrics

• commit acceptance rate based on gender, geographic location, use of language (foreign tongue). – is an indicator for welcoming culture and potential for diversity
• group think – diversity indicator
• monoculture – don’t know how to measure that
• existence of a code of conduct – makes culture explicit
• sentiment and retention (contributor life cycle, turn-over rate)
• aggregate through include-tree: Age of an old included library might compromise entire project.
• contributions outside of code (e.g. providing testing environments and running tests)
• documentation (wiki activity)
• stack overflow numbers
• identify strong community supporters. (e.g. who is answering most emails, who is closing most issues, who is replying to most stack overflow questions…)
  • diversity of people responding to community questions
• track new developers (path to leadership)
• aggregate from repositories, to projects, to community (with drill down features) – providing the ability to go from aggregate data to granular data to identify issues. (not only report medians, means, but also standard deviation, or quartile-spread) – “composite numbers hide a lot of details!”
• maturity label provided by a community or foundation (incubator vs. mature)

Problem with metrics is that people will feel tracked and react to the metrics. Communities might focus more on the numbers than on great software, splitting up commits, replying nonsensically on issues, etc... to drive metrics. Idea: drop term “metric” and call it “community insight”