BUG: False negative for Security Policy check on [Laravel](https://github.com/laravel/framework) project

[diogoteles08]

Describe the bug The Scorecards check for Security Policy did not find any Security Policy file on Laravel project, but it does have a SECURITY.md file inside .github folder.

Reproduction steps Steps to reproduce the behavior:

1. Run Scorecards CLI for the repo https://github.com/laravel/framework, or access the already evaluated result here
2. Note that for Security-Policy check it's got a null punctuation with reason "security policy file not detected"
3. Note that the repository actually have a security policy. It's located on .github/SECURITY.md, which is a location considered by GitHub itself, as you can check here.

Expected behavior Scorecards should detect the SECURITY.md file and use its content to evaluate the punctuation for Security-Policy check.

[spencerschrock]

So the tarball downloaded by hitting https://api.github.com/repos/laravel/framework/tarball/ doesn't contain the .github folder at all, which is why scorecard isn't picking it up. The scorecard repo downloaded via the equivalent link does include the .github folder for example

I'm not familiar with the tarball endpoint, and the GitHub docs don't seem to explain the behavior.

/tmp$ tar -tvf laravel-framework-v9.41.0-25-gdefd920.tar.gz | grep .github
/tmp$

[laurentsimon]

That's an interesting finding. Other checks look for files under .github workflow (Token-Permissions, Dangerous-Workflows, Pinned-Dependencies). Does this affect the other checks too for this repository?

[gabibguti]

@laurentsimon Yes. Scorecards gives a 10 for Token-Permissions, when they do not set it as Scorecards expects, and also a 10 for Pinned-Dependencies, when they do not hash-pin the dependencies and for some not even tag-pin. So, if the TAR comes "wrong" from GH API, these checks that look at the repo's source code will be mistaken.

[spencerschrock]

This is a configuration issue. By excluding the .github folder in their .gitattributes file, it's unavailable for analysis by scorecard https://github.com/laravel/framework/blob/926cf9686c28ea6424990e2bd36dd607695eb104/.gitattributes#L9

[azeemshaikh38]

Great find @spencerschrock! Maybe #1709 could help here?

[diogoteles08]

Great find indeed! Would help if I close this issue and create a different one focusing on this main cause?

[azeemshaikh38]

Let's keep this open. This bug has all the context needed for someone in the future.