BUG - Branch Protection check data in cron

[naveensrinivasan]

Describe the bug I took a sample of 300+ repositories and Branch-Protection is failing in all of them. Are we sure this check is working?

This is failing for scorecard also

SELECT h.Confidence,h.Name,h.Pass,Repo FROM `openssf.scorecardcron.scorecard`, UNNEST(checks) as h WHERE Repo in ("github.com/ossf/scorecard") and h.Name = "Branch-Protection"

[naveensrinivasan]

There are only 3 repositories that are passing with this check.

SELECT h.Confidence,h.Name,h.Pass,Repo,Date FROM `openssf.scorecardcron.scorecard`, UNNEST(checks) as h WHERE 
h.Pass = true and 
h.Name = "Branch-Protection"

[
  {
    "Confidence": "10",
    "Name": "Branch-Protection",
    "Pass": true,
    "Repo": "github.com/googleinterns/play-web-step-2020",
    "Date": "2021-08-30"
  },
  {
    "Confidence": "10",
    "Name": "Branch-Protection",
    "Pass": true,
    "Repo": "github.com/googleinterns/step18-2020",
    "Date": "2021-09-20"
  },
  {
    "Confidence": "10",
    "Name": "Branch-Protection",
    "Pass": true,
    "Repo": "github.com/googleinterns/play-web-step-2020",
    "Date": "2021-09-06"
  }
]

[azeemshaikh38]

I think this is expected because of https://github.com/ossf/scorecard/blob/main/checker/check_result.go#L31

We base our Pass/Fail based on score now, so since most repos will not have score > 8 for Branch-Protection we end up failing.

[naveensrinivasan]

So we can never determine if the Branch-Protection is on without a passing score?

[azeemshaikh38]

Yes, a complex check like Branch-Protection which looks at many different settings/options, will have that problem with both Fail/Pass model and the 1-10 score model. Even with a passing score, we cannot be sure that the "right" settings are enabled to give it this score. It's not ideal, but it's the best we can do right now.

In future, our hope is that the Scorecard policies can improve these complex checks by providing granularity - https://docs.google.com/document/d/19PzTxjVL9iR6tEkCz_wPX2-xLJQtWzPSDxG8GZPu0Ao/edit?resourcekey=0-y_cKeath-Yq9TBTtKy7pZA

[azeemshaikh38]

Closing the issue, re-open if needed.