Add pre-commit hook that detects secrets

[r0mant]

Let's add a git pre-commit hook that prevents folks from accidentally pushing secrets to Github.

Research/investigate what options there are and implement this.

[codeknight03]

Looks like an interesting quest though looks like it has already been assigned would like to be part of the research or implementation.

[doggydogworld]

After doing some investigating I found two tools that are well supported and open source.

• gitleaks (Go, 17k stars, MIT License)
• trufflehog (Go, 14k stars, AGPL-3.0 License)

Just to compare them from a UX standpoint

Features

• gitleaks
  • 160 supported secret types. Secret types are determined via regex pattern matches. List here
  • Supports filesystem scan, git history scans, and pre-commit staging scans.
  • Supports configuration of custom pattern matching rules
  • Noise reduction:
  • Allows a baseline scan to be added and checked against. All future hits that are in the baseline are ignored.
  • Applies fingerprint to each credential found that can selectively be ignored by adding to a .gitleaksignore
  • For data that allows comments a gitleaks:allow line can be added to flag the secret as safe.
  • Configured by default to ignore common noisy filepaths (node_modules, .git, go.mod, etc). No intuitive to configure though.
• trufflehog
  • Over 700 supported secret types via regex pattern matches.
  • verified-only mode
  • Attempt to use credentials against matched APIs to determine if secrets are "active" (e.g. AWS token found will be attmpted against GetCallerIdentity resource)
  • Supports custom pattern matching and custom verified detection rules (i.e. to test against internal APIs)
  • Supports filesystem scan, git history scans, and pre-commit staging scans.
  • Noise reduction:
  • verified mode can reduce false positives by only reporting active secrets.
  • Optional exclude directories. No default configuration though so for example node_modules gives a lot of hits.
  • For data that allows comments a trufflehog:ignore line can be added to flag the secret as safe.

Quick Testing

Created a fake SSH key, TLS private/public key pair, auto-generated password, and canary AWS credentials.

Also did a quick test against all files in the teleport repo to check noise levels and check the upper bound of the execution time. The check did not include git history only the files present in the latest commit.

Neither caught the auto-generated password but this could be configured in both to be more sensitive.

• gitleaks
  • pre-commit had 3 hits (SSH, TLS, AWS)
  • Scan against whole repo had 100 hits
  • Execution time:
  • pre-commit: ~15ms
  • Whole repo: ~6 seconds
• trufflehog
  • pre-commit had 3 hits (SSH, TLS, AWS)
  • pre-commit with verified-only enabled only 1 hit (canary AWS creds)
  • Scan against whole repo had 500 hits (0 verified)
  • Execution time:
  • pre-commit: ~15 seconds
  • whole repository: ~1min

Conclusion

We should use gitleaks for pre-commit hooks.

While trufflehog is very powerful and has a lot of really nice features for reducing noise that does come at a cost to execution time. With the extremely small commit I tested it against it took almost 15 seconds to run. It's important to also recognize that in verified mode it establishes connections which depending on network conditions could slow it down even more. It might be a good idea to look into trufflehog for frequent scans post-commit but it could be too intrusive in the development process as a pre-commit hook.

[r0mant]

@doggydogworld I agree with your assessment, let's use gitleaks.

[r0mant]

We've decided to not do this for the time being given hosted Github does not support server-side pre-commit hooks and making this feature opt-in makes it far less effective. Github's builtin secret scanning on push should do fine for now.