Feature: Improve dependabot detection thru PRs or parsing config file

ossf / scorecard

OpenSSF Scorecard - Security health metrics for Open Source

https://scorecard.dev

Apache License 2.0

4.47k stars 489 forks source link

Feature: Improve dependabot detection thru PRs or parsing config file #874

Closed arieltorti closed 2 years ago

arieltorti commented 3 years ago

Describe the bug When running Scorecard against this repo https://github.com/tngan/samlify I get an score 0 on the Dependency Update Tool check, however the repo does have dependabot.

Reproduction steps Steps to reproduce the behavior:

Run docker run -e GITHUB_AUTH_TOKEN=<token> gcr.io/openssf/scorecard:stable --show-details --repo=https://github.com/tngan/samlify
Check the score of the Dependency Update Tool.

Expected behavior Dependency Update Check has a non-zero score because the repo has dependabot.

laurentsimon commented 3 years ago

do you have dependabot enabled as a GitHub setting rather than a config file under .github/?

arieltorti commented 3 years ago

I'm not the owner of the samlify repository so I can't fully confirm.

But after some testing on a private repo I own I noticed that dependabot is not detected if enabled through Github settings, I would suspect same thing is happening with samlify.

laurentsimon commented 3 years ago

yep that makes sense. We encourage owners to make their config public.

That said, we have plans to improve the check to look for dependabot PRs (accepted/rejected) to make the check more robust. Thanks for reminding us, I realized there was no tracking issue for it.

If you're interested in helping, PRs are welcome!

laurentsimon commented 3 years ago

Additional note: we should also check for reverted dependabot PRs by maintainers

arieltorti commented 3 years ago

That said, we have plans to improve the check to look for dependabot PRs (accepted/rejected) to make the check more robust.

Any ideas on how to implement this ? I've been looking at the Github API and haven't found a way to know if the repository has dependabot (or similar) enabled. Another way would be to check for commits authored by the dependabot in the last N days, what do you think about it ?

laurentsimon commented 3 years ago

That said, we have plans to improve the check to look for dependabot PRs (accepted/rejected) to make the check more robust.

Any ideas on how to implement this ? I've been looking at the Github API and haven't found a way to know if the repository has dependabot (or similar) enabled. Another way would be to check for commits authored by the dependabot in the last N days, what do you think about it ?

I think that is a good start and aligned with what we had in mind. Do you know if this API https://docs.github.com/en/rest/reference/pulls#check-if-a-pull-request-has-been-merged can list PRs that have been rejected/dismissed as well?

@jeffmendoza please let us know if you have better ideas/insights for now.

FYI @inferno-chromium @naveensrinivasan @oliverchang @azeemsgoogle how many days shall we look back for merged PRs?

laurentsimon commented 2 years ago

see also a discussion https://github.com/step-security/agent/issues/35#issuecomment-974584647, between dependabot security and dependabot dep. We should check whether the config file can differentiate between the two types of dependabot.

FYI @varunsh-coder

laurentsimon commented 2 years ago

(renaming the title of this issue to Feature: Improve dependabot detection thru PRs or parsing config file)

azeemshaikh38 commented 2 years ago

Looking at PRs to get a hint about dependabot usage sounds like a good idea to me. I would say a good starting point would be to use the existing ListMergedPRs API. We can then consider expanding the support to non-merged or reverted PRs too.