search

edoardottt / cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
https://edoardoottavianelli.it
GNU General Public License v3.0
1.49k stars 152 forks source link

Add trufflehog for secret detection #150

Open hugo-syn opened 4 months ago

hugo-syn commented 4 months ago

Hi,

I've discovered your tool via secator it's very nice thanks for your work !

In this PR I'm adding 2 things, the first big improvement is the use of trufflehog for secret detection. Trufflehog can detect a looot of credentials, the complete list can be found here. This change make the tool slower for secret detection but improve the secret detection.

I've also add a command line option to ignore some url for secret scaning. For example scaning png files for secrets can be avoided:

$ cariddi  -s -secret-extension-filter pdf,svg,png,jpeg
...
edoardottt commented 4 months ago

Hi @hugo-syn , thank you so much for improving cariddi! Appreciated :)

hugo-syn commented 4 months ago

Hi, @edoardottt I've fixed the merge conflicts, I've updated some packages.

For the PR I understand, don't hesitate if you have questions or remarks regarding my code :)

edoardottt commented 4 months ago

If possible (there could be issues with deps) revert back go.mod to using go 1.21 and remove toolchain info (the linter action fails :,( ).

hugo-syn commented 4 months ago

The trufflehog package requires go 1.22.0:

$ go build ./cmd/cariddi
go: github.com/trufflesecurity/trufflehog/v3@v3.73.0 requires go >= 1.22.0 (running go 1.21.0)

I can bump the golang version in the different CI files if you want

hugo-syn commented 4 months ago

And I've checked it looks good with go 1.22.0

edoardottt commented 4 months ago

Hi @hugo-syn,

And I've checked it looks good with go 1.22.0

the go build action is failing:

go build -v ./...
  shell: /usr/bin/bash -e {0}
  env:
    GOROOT: /opt/hostedtoolcache/go/1.18.10/x64
go: errors parsing go.mod:
/home/runner/work/cariddi/cariddi/go.mod: invalid go version '1.22.0': must match format 1.23
/home/runner/work/cariddi/cariddi/go.mod: unknown directive: toolchain

https://github.com/edoardottt/cariddi/actions/runs/8754239537/job/24025715013?pr=150#step:6:8

I think this is caused by github actions, they must be set to the correct go version too (all of them: go, linting...).

hugo-syn commented 4 months ago

I've change the golang version in the workflows to 1.22.0 the build pipeline is ok now but there are errors with the golang-ci

edoardottt commented 4 months ago

I've change the golang version in the workflows to 1.22.0 the build pipeline is ok now but there are errors with the golang-ci

I've seen. See if locally everything is okay with golangci-lint run. If you need help I'm here. In the next days I'm gonna review it. Since it's a huge PR I guess there will be a lil bit of work to be done... :/ Let's see.

This change make the tool slower

Can you quantify this? I'd like to comprehend how much this will be slower.

cc @hugo-syn . Let me know if you need assistance :)

hugo-syn commented 3 months ago

Hi @edoardottt Unfortunately I didn't manage to fix the CI there are new errors with code that I did not modify :/

Can you quantify this? I'd like to comprehend how much this will be slower.

It's difficult to say but since it runs all the secret detection of trufflhog the secret search will take more time but it will find lots of new secrets

edoardottt commented 3 months ago

Unfortunately I didn't manage to fix the CI there are new errors with code that I did not modify :/

Ok, so don't worry about golangci-lint errors. I'll fix them. Go build is the important one and it's not failing.

It's difficult to say but since it runs all the secret detection of trufflhog the secret search will take more time but it will find lots of new secrets

Ok, I'll merge in devel and perform some tests.

For now it's okay so, wait for my PR review :)

hugo-syn commented 3 months ago

Do not hesitate if you have any questions ! :)

hugo-syn commented 1 month ago

Hi @edoardottt I'm not used to dev practices what should I do ? feel free to rewrite what I did to fit your way :)

edoardottt commented 1 month ago

Hi @edoardottt I'm not used to dev practices what should I do ? feel free to rewrite what I did to fit your way :)

Hi @hugo-syn . I don't have write access to your fork + this PR is huge + it seems it was created a while ago and since this would be a big improvement I'll proceed in this way: I'll create a PR by myself to merge in devel and you can review that for me. Maybe this will speed up the process.

hugo-syn commented 1 month ago

Oh ok I thought there would be a mechanism so that you could modify the code directly. Sounds good to me. Do not hesitate if you have any question