Scan GitHub and GitLab refs that aren't pulled by default #1588

Open rgmz opened 1 year ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

This article featured on HackerNews reminded me that both GitHub and GitLab have refs for (merge|pull) requests that — to my knowledge — aren't pulled by default. This is especially juicy for TruffleHog because PRs may contain commits or alternate history with valid secrets that technically no longer exist in the repository.

GitHub

$  git clone https://github.com/trufflesecurity/trufflehog.git
$ cd trufflehog
$ git ls-remote
From https://github.com/trufflesecurity/trufflehog.git
69021f59c57b6a49bafcc8d1827e355405330014        HEAD
aa3ba817175e1bcdf852f11ffa327c1d134ecb26        refs/heads/0x1/add-on-prem-verification-flag
ade5d91d5ca94e996377b1909a802dd9dbc51b6b        refs/heads/1560-error-during-git-scan-using-pre-commit
...
c3644ccaf1bb0799ef919a5897b7737ebff428db        refs/pull/1/head
180b9c288bfc2b0f0543f91dfd74740f4f1d410e        refs/pull/100/head
c36defae7818b738389815dd32a64db398e2d833        refs/pull/100/merge
...
3048c6429d80094b42fd44919e3d6768536178ce        refs/tags/2.0.97
6d2dc5e965f9d5c23acea0c0c6377b06a8ae6c7c        refs/tags/v0.1.0-alpha
fda044631b344997a4556f52aadbd7c8275d0802        refs/tags/v3.0.0

GitLab

$ git clone git@gitlab.com:pdftk-java/pdftk.git
$ cd pdftk
$ git ls-remote
From git@gitlab.com:pdftk-java/pdftk.git
be326bcca502dcdcbfd6dc63f31976ea5aa7ffc9        HEAD
0d30918c940cae11e48aeb9c86857bb1455fd52a        refs/heads/ci
6c6fccca904b9234fa20e5c372e4573c0442156d        refs/heads/ci_native
...
53680180fef65bc6cd9fedc90b345ef354ab8c2c        refs/merge-requests/1/head
a86eca79300f9b0db0d2af2b6cb411168e8df16a        refs/merge-requests/12/head
696c8d99b25552123344bda752d3df6b8929adab        refs/merge-requests/12/merge
...
e80603a9130240019417eec30d7edad6cecba325        refs/tags/v3.0.0
4e20310b472b52cca12ad94252f90cff6cfebad6        refs/tags/v3.0.1

Others

[ ] Bitbucket
[ ] Gitea
[ ] Gogs
[ ] Sourcehut
[ ] Forgejo
[ ] ?

Problem to be Addressed

Find valid secrets that may be hidden in old PR histories.

Description of the Preferred Solution

Off the top of my head, I do not have any ideas on how to tackle this. It seems that you need to manually query for (git ls-remote) and checkout each ref in a detached state. The history will not show up via git log otherwise.

Scanning refs could result in the same commits being scanned twice. I am not sure if this would produce duplicate results, or if we already deduplicate results.

Additional Context

N/A

References

N/A

This seems trivial to implement:

# assuming the origin is GitHub, which can be checked with `git ls-remote`
git fetch origin "+refs/pull/*:refs/heads/pull/*"

can this behaviour be confirmed to be present in TruffleHog?

trufflesecurity / trufflehog