Review hipcheck for potential criteria & code improvements (feature request)

[david-a-wheeler]

Is your feature request related to a problem? Please describe.

MITRE's hipcheck attempts to analyze OSS to measure security, with a concept similar to Scorecard. I expect that hipcheck has some capabilities not currently in Scorecard.

Describe the solution you'd like

Review of hipcheck criteria and code to see what can be gleaned from it. hipcheck is released under Apache-2.0.

\I imagine there may be derivative issues and/or PRs generated eventually from this.

[github-actions[bot]]

Stale issue message - this issue will be closed in 7 days

[distractible]

Note that the Hipcheck team are interested in this: https://github.com/mitre/hipcheck/commit/43723903d66cd722c3d7e25265fdcc29b98b537d

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity.

[spencerschrock]

In October I briefly looked at MITRE's Hipcheck v3.1.0-alpha.0 and tried to map it to an existing Scorecard check. The understanding of Hipcheck analyses comes from their documentation summary here.

New Analyses

These analyses don’t have a comparable check in Scorecard. Some could be implemented now, some require better git support in Scorecard, and some would be API intensive but doable.

• Churn (Repo) Each commit gets a score, comparing its size to the average commit size from that repo. The goal is to flag suspicious commits with the rationale that: “it's easier to hide malicious content in a large commit than in a small one”.
• Entropy (Repo) Each commit gets a score, based on its textual randomness. Again, flagging suspicious commits under the rationale that “high textual randomness, may indicate the presence of packed malware or obfuscated code”.
• Typo (Repo) Parses a repo’s npm dependencies, and compares against popular libraries to look for typosquatting.
• Contributor Trust (PR) Defining a contributor by their email, determines if a contributor is trusted by looking for M commits in N recent months.
• Module Contributors (PR) Look for contributors who are modifying part of the code (at the granularity of JavaScript modules) for the first time.

Related Analyses

These analyses do similar things to a Scorecard check, but the semantics differ.

• Affiliation (Repo and PR) Similar to the Contributors check from Scorecard, this analysis looks at contributors using their email addresses for attribution. While Scorecard looks for contributions from multiple companies, Hipcheck compares contributors against an allow/deny list on the basis of company or country.

Identical Analyses

These analyses are generally a direct match to an existing Scorecard check. While there may be some differences in detection or scoring, these are already considered part of Scorecard.

• Activity (Repo) The Scorecard analog is the Maintained check. Slight differences are present in the lookback period (Hipcheck 1 year, Scorecard 90 days). It also doesn’t take into account archive status or issue activity. Hipcheck also notes that this can be a noisy heuristic for software that is “done”, which is a problem Scorecard has seen.
• Binary (Repo) Almost identical to Binary-Artifacts from Scorecard. The threshold and detection have slight differences.
• Fuzz (Repo) A subset of Scorecard’s Fuzzing check, which looks for OSS Fuzz integration.
• Identity (Repo) Identical to the implicit portion of Scorecard’s Code-Review, which compares the identities of who authors and merges/commits code. Hipcheck uses git data, while Scorecard uses GitHub/GitLab identities.
• Review (Repo) Identical to the portion of Code-Review which uses GitHub’s formal PR review system.

[LappleApple]

@spencerschrock @afmarcum and others: From our Jan 25 meeting, we said we'd close this one as it's "already done."

[spencerschrock]

I imagine there may be derivative issues and/or PRs generated eventually from this.

I think any relevant derivative issues would still need to be opened first, based on the analysis.

[github-actions[bot]]

This issue has been marked stale because it has been open for 60 days with no activity.