Add trufflehog for secret detection

[hugo-syn]

Hi,

I've discovered your tool via secator it's very nice thanks for your work !

In this PR I'm adding 2 things, the first big improvement is the use of trufflehog for secret detection. Trufflehog can detect a looot of credentials, the complete list can be found here. This change make the tool slower for secret detection but improve the secret detection.

I've also add a command line option to ignore some url for secret scaning. For example scaning png files for secrets can be avoided:

$ cariddi  -s -secret-extension-filter pdf,svg,png,jpeg
...

[edoardottt]

Hi @hugo-syn , thank you so much for improving cariddi! Appreciated :)

• I don't have a lot of time and since this is a big PR I'll take some time to review it.
• For now try to resolve the conflicts with the devel branch for go.mod and go.sum files.

[hugo-syn]

Hi, @edoardottt I've fixed the merge conflicts, I've updated some packages.

For the PR I understand, don't hesitate if you have questions or remarks regarding my code :)

[edoardottt]

If possible (there could be issues with deps) revert back go.mod to using go 1.21 and remove toolchain info (the linter action fails :,( ).

[hugo-syn]

The trufflehog package requires go 1.22.0:

$ go build ./cmd/cariddi
go: github.com/trufflesecurity/trufflehog/v3@v3.73.0 requires go >= 1.22.0 (running go 1.21.0)

I can bump the golang version in the different CI files if you want

[hugo-syn]

And I've checked it looks good with go 1.22.0

[edoardottt]

Hi @hugo-syn,

And I've checked it looks good with go 1.22.0

the go build action is failing:

go build -v ./...
  shell: /usr/bin/bash -e {0}
  env:
    GOROOT: /opt/hostedtoolcache/go/1.18.10/x64
go: errors parsing go.mod:
/home/runner/work/cariddi/cariddi/go.mod: invalid go version '1.22.0': must match format 1.23
/home/runner/work/cariddi/cariddi/go.mod: unknown directive: toolchain

https://github.com/edoardottt/cariddi/actions/runs/8754239537/job/24025715013?pr=150#step:6:8

I think this is caused by github actions, they must be set to the correct go version too (all of them: go, linting...).

[hugo-syn]

I've change the golang version in the workflows to 1.22.0 the build pipeline is ok now but there are errors with the golang-ci

[edoardottt]

I've change the golang version in the workflows to 1.22.0 the build pipeline is ok now but there are errors with the golang-ci

I've seen. See if locally everything is okay with golangci-lint run. If you need help I'm here. In the next days I'm gonna review it. Since it's a huge PR I guess there will be a lil bit of work to be done... :/ Let's see.

This change make the tool slower

Can you quantify this? I'd like to comprehend how much this will be slower.

cc @hugo-syn . Let me know if you need assistance :)

[hugo-syn]

Hi @edoardottt Unfortunately I didn't manage to fix the CI there are new errors with code that I did not modify :/

Can you quantify this? I'd like to comprehend how much this will be slower.

It's difficult to say but since it runs all the secret detection of trufflhog the secret search will take more time but it will find lots of new secrets

[edoardottt]

Unfortunately I didn't manage to fix the CI there are new errors with code that I did not modify :/

Ok, so don't worry about golangci-lint errors. I'll fix them. Go build is the important one and it's not failing.

It's difficult to say but since it runs all the secret detection of trufflhog the secret search will take more time but it will find lots of new secrets

Ok, I'll merge in devel and perform some tests.

For now it's okay so, wait for my PR review :)

[hugo-syn]

Do not hesitate if you have any questions ! :)

[hugo-syn]

Hi @edoardottt I'm not used to dev practices what should I do ? feel free to rewrite what I did to fit your way :)

[edoardottt]

Hi @edoardottt I'm not used to dev practices what should I do ? feel free to rewrite what I did to fit your way :)

Hi @hugo-syn . I don't have write access to your fork + this PR is huge + it seems it was created a while ago and since this would be a big improvement I'll proceed in this way: I'll create a PR by myself to merge in devel and you can review that for me. Maybe this will speed up the process.

[hugo-syn]

Oh ok I thought there would be a mechanism so that you could modify the code directly. Sounds good to me. Do not hesitate if you have any question

[ocervell]

@hugo-syn great idea to add trufflehog ! @edoardottt for what it's worth, maintainers can make changes on PRs using the GitHub CLI by running gh pr checkout 150

[edoardottt]

Hi @ocervell ! The changes were applied in https://github.com/edoardottt/cariddi/tree/trufflehog. However it produces a lot of false positives, it's far than acceptable.