Github Health Categories and Metrics Suggestions

[GeorgLink]

From @Ashkeelun on January 26, 2017 15:8

Hello Everyone,

In working with @germonprez in his class at the University of Nebraska Omaha, I was asked to identify categories and some metrics associated with them. He suggested posting this here for discussion.

When presented with this problem I first took a step back and looked at how repositories flourish on Github. It is important to note that the code itself is not the only thing of concern. The goal here is to address and rank repositories on what the user is concerned about. From this I derive five key areas that I named: • Community – Active contributors to the Repository and their growth and activity. • Code – Quality and reliability of the source code itself. • Assistance – Quality and helpfulness of issue resolution. • Adaptability – The ability for the code to have a variety of uses. • Licensing – Usability of the code.

It is important to segregate these concerns and have the ability to judge them separately as this should help diversify the systems adaptiveness to varying concerns. An entity that only plans on using source internally may not need to consider the license but is concerned about the quality of the code and community to support that code.

Community refers to the active contributors that support a repository. Looking at the activity of each contributor both on the repository and then on Github in general to determine how much time they commit to the repo compared to other projects. This should also look at the interaction, closeness and growth of the contributors over time. A few metrics that would apply to this are: • number of contributors • frequency of contributions • activity level of contributors • Truck Factor (“The number of developers it would need to lose to destroy its progress.” From the Influence Analysis of Github Repositories) • Time to become contributor • Distribution of work across community • rate of acceptance of new contributions

Code is probably the easiest category and the hardest category to evaluate. Ideally, we want to know that it is routinely kept up to date, is clean and well documented, and will continue to stay that way for the foreseeable future. This is easier said than done, one thing that makes it easier to analyze is the fact that it has the most meta data to work with. A few metrics that would apply are: • number of updates • regularity of updates • time since last update • number of pull rejections • number of CVEs and % of age open – https://www.coreinfrastructure.org/programs/census-project • 3rd party Dependencies (if obtainable) – https://www.coreinfrastructure.org/programs/census-project • stars • overall size of the repository / commits

Assistance is exactly what it sounds like. As a user of the code how much assistance can you get in implementing it. Additionally, while this may not be directly relevant to some entities, it is indirectly relevant to everyone. Lack of support leads to lower adoption which also leads to a smaller set of stakeholders willing to keep it going. A few metrics that would apply are: • number of open issues / label • time to close • communication level (response time and number)

Adaptability refers to the degree that the project could be easily adapted to your specific needs. While this is very useful it’s also extremely hard to determine from the metrics. However, I believe a couple could give small indirect indications of flexibility. The first is the number of forks in the repository followed by the number of downloads. A large number of forks with lower downloads tend indicate a useful code that can be expanded upon in many ways. Where a low number of forks but large number of downloads may indicate a project that is specific but widely useful. More research will need to be done to identify and refine these assumptions.

Licensing which is in reference to the usability of the code. More restrictive licenses may be a turn off or may just require more adaptability and community to be viable. A couple metrics for licenses would be: • Is there a license • Number of licenses • Flexibility of licenses

Copied from original issue: OSSHealth/ghdata#7

[GeorgLink]

From @abuhman on January 26, 2017 20:59

Thanks, these are some good ideas!

[GeorgLink]

Thanks @Ashkeelun ,

I moved the issue to the https://github.com/OSSHealth/HealthIndicators repository because it is for the discussion and development of the metrics.

[abuhman]

Another category that was mentioned during our meeting was compliance, which to my understanding was related to risk/security.

[GeorgLink]

@abuhman I agree with you. Compliance is related to risk/security - maybe they are the same category.

[germonprez]

Yes. The general thinking is

Risk

• License and Copyright
• Vulnerabilities (CVE)
• Weaknesses (CWE)

[abuhman]

I've found some papers on the subject of calculating truck/bus factor. If we want to include truck factor as a metric, they might be a place to start. So far, though, I have not found a standard way to calculate truck factor.

Links to papers:

A short paper that includes a description on how the authors calculated truck factor based on previous research https://peerj.com/preprints/1233v1.pdf

Two longer and more detailed papers developing the calculations used above: http://delivery.acm.org/10.1145/2520000/2512207/a14-fritz.pdf?ip=137.48.185.192&id=2512207&acc=ACTIVE%20SERVICE&key=70F2FDC0A279768C%2EB2398D6700DCDCF5%2E4D4702B0C3E38B35%2E4D4702B0C3E38B35&CFID=722612431&CFTOKEN=63073148&__acm__=1485801520_c29634c81a065733b77983160ebb8cbf http://delivery.acm.org/10.1145/1810000/1806856/p385-fritz.pdf?ip=137.48.185.192&id=1806856&acc=ACTIVE%20SERVICE&key=70F2FDC0A279768C%2EB2398D6700DCDCF5%2E4D4702B0C3E38B35%2E4D4702B0C3E38B35&CFID=722612431&CFTOKEN=63073148&__acm__=1485803377_517ec1426d6732bfcae9a0f2af09d071

An alternative way of calculating bus, which mentions a tool created by the authors to calculate bus factor: http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7081864

Another alternative, though the authors emphasize it is preliminary: http://delivery.acm.org/10.1145/1990000/1985379/p12-torchiano.pdf?ip=137.48.185.192&id=1985379&acc=ACTIVE%20SERVICE&key=70F2FDC0A279768C%2EB2398D6700DCDCF5%2E4D4702B0C3E38B35%2E4D4702B0C3E38B35&CFID=722612431&CFTOKEN=63073148&

[GeorgLink]

Thank you. I'll have a look at these too.

[abuhman]

A note on the original post since I wasn't aware: CVE stands for Common Vulnerabilities and Exposures More information: https://cve.mitre.org/about/

[GeorgLink]

The Links in the original post don't work anymore. Below are the papers I think @abuhman originally posted.

Cosentino, V., Izquierdo, J. L. C., & Cabot, J. (2015). Assessing the bus factor of Git repositories. In 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER) (pp. 499–503). https://doi.org/10.1109/SANER.2015.7081864

Fritz, T., Murphy, G. C., Murphy-Hill, E., Ou, J., & Hill, E. (2014). Degree-of-knowledge: Modeling a Developer’s Knowledge of Code. ACM Trans. Softw. Eng. Methodol., 23(2), 14:1–14:42. https://doi.org/10.1145/2512207

Fritz, T., Ou, J., Murphy, G. C., & Murphy-Hill, E. (2010). A Degree-of-knowledge Model to Capture Source Code Familiarity. In Proceedings of the 32Nd ACM/IEEE International Conference on Software Engineering - Volume 1 (pp. 385–394). New York, NY, USA: ACM. https://doi.org/10.1145/1806799.1806856

Torchiano, M., Ricca, F., & Marchetto, A. (2011). Is My Project’s Truck Factor Low?: Theoretical and Empirical Considerations About the Truck Factor Threshold. In Proceedings of the 2Nd International Workshop on Emerging Trends in Software Metrics (pp. 12–18). New York, NY, USA: ACM. https://doi.org/10.1145/1985374.1985379