ossf / scorecard

OpenSSF Scorecard - Security health metrics for Open Source
https://scorecard.dev
Apache License 2.0
4.63k stars 503 forks source link

"internal error: cannot list check runs by ref" with the bcc repository #1285

Closed evverx closed 2 years ago

evverx commented 3 years ago

Looks like https://github.com/iovisor/bcc/pull/3626 confuses scorecard somehow

./scorecard --verbosity Debug  --show-details --format json  --repo=https://github.com/iovisor/bcc --checks CI-Tests
...
{
  "date": "2021-11-15",
  "repo": {
    "name": "github.com/iovisor/bcc",
    "commit": "67f59ee80fcf5deedaacba1436d9fa09d32a16a0"
  },
  "scorecard": {
    "version": "3.1.1-64-gcc49494-dirty",
    "commit": "cc4949465b6730ee398e49a096e0132f02078372"
  },
  "score": -1.0,
  "checks": [
    {
      "details": [
        "Debug: checking PR: 3626"
      ],
      "score": -1,
      "reason": "internal error: cannot list check runs by ref",
      "name": "CI-Tests",
      "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/cc4949465b6730ee398e49a096e0132f02078372/docs/checks.md#ci-tests",
        "short": "Determines if the project runs tests before pull requests are merged."
      }
    }
  ],
  "metadata": null

To get that "Debug: checking PR: 3626" message I applied the following patch:

diff --git a/checks/ci_tests.go b/checks/ci_tests.go
index 515cbb1..a84d984 100644
--- a/checks/ci_tests.go
+++ b/checks/ci_tests.go
@@ -53,6 +53,9 @@ func CITests(c *checker.CheckRequest) checker.CheckResult {

                var foundCI bool

+               c.Dlogger.Debug3(&checker.LogMessage{
+                       Text: fmt.Sprintf("checking PR: %d", pr.Number),
+               })
                // Github Statuses.
                prSuccessStatus, err := prHasSuccessStatus(pr, c)
                if err != nil {
evverx commented 3 years ago

https://github.com/systemd/casync triggers it too

    {
      "details": null,
      "score": -1,
      "reason": "internal error: cannot list check runs by ref",
      "name": "CI-Tests",
      "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/bd9b6df2d372718aebe7f1fb4c2a6c32149e5868/docs/checks.md#ci-tests",
        "short": "Determines if the project runs tests before pull requests are merged."
      }
    },