trufflesecurity / trufflehog

Find, verify, and analyze leaked credentials
https://trufflesecurity.com
GNU Affero General Public License v3.0
17.19k stars 1.71k forks source link

Multiple data races detected with `go build -race` #3027

Open rgmz opened 4 months ago

rgmz commented 4 months ago

Please review the Community Note before submitting

TruffleHog Version

HEAD

Steps to Reproduce

  1. git clone https://github.com/trufflesecurity/trufflehog.git
  2. go build -race
  3. ./trufflehog github --repo="https://github.com/trufflesecurity/trufflehog.git" --only-verified

Description

Go reports 2084 data races in this particular instance. I've grouped and condensed a few of the main causes into <details> blocks for the sake of legibility, however, this may not tell the entire picture.

1. overseer / main.go (click here to expand) ``` ================== WARNING: DATA RACE Read at 0x00c000f46900 by goroutine 80: github.com/jpillora/overseer.(*parent).handleSignal() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:135 +0x1da github.com/jpillora/overseer.(*parent).setupSignalling.func1() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:113 +0x4b Previous write at 0x00c000f46900 by main goroutine: github.com/jpillora/overseer.(*parent).fork() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:367 +0x111 github.com/jpillora/overseer.(*parent).forkLoop() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:356 +0x329 github.com/jpillora/overseer.(*parent).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:65 +0x339 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c Goroutine 80 (running) created at: github.com/jpillora/overseer.(*parent).setupSignalling() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:111 +0x1b0 github.com/jpillora/overseer.(*parent).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:56 +0x1e4 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c ================== ================== WARNING: DATA RACE Read at 0x00c0000176a0 by goroutine 80: github.com/jpillora/overseer.(*parent).handleSignal() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:135 +0x21a github.com/jpillora/overseer.(*parent).setupSignalling.func1() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:113 +0x4b Previous write at 0x00c0000176a0 by main goroutine: os/exec.Command() /home/user/sdk/go1.22.1/src/os/exec/exec.go:377 +0x145 github.com/jpillora/overseer.(*parent).fork() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:364 +0xf5 github.com/jpillora/overseer.(*parent).forkLoop() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:356 +0x329 github.com/jpillora/overseer.(*parent).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:65 +0x339 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c Goroutine 80 (running) created at: github.com/jpillora/overseer.(*parent).setupSignalling() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:111 +0x1b0 github.com/jpillora/overseer.(*parent).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:56 +0x1e4 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c ================== ================== WARNING: DATA RACE Read at 0x00c001929ad0 by goroutine 80: os.(*Process).signal() /home/user/sdk/go1.22.1/src/os/exec_unix.go:61 +0x53 os.(*Process).Signal() /home/user/sdk/go1.22.1/src/os/exec.go:140 +0x107 github.com/jpillora/overseer.(*parent).sendSignal() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:161 +0xa8 github.com/jpillora/overseer.(*parent).handleSignal() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:144 +0x3a4 github.com/jpillora/overseer.(*parent).setupSignalling.func1() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:113 +0x4b Previous write at 0x00c001929ad0 by main goroutine: os.newProcess() /home/user/sdk/go1.22.1/src/os/exec.go:29 +0x729 os.startProcess() /home/user/sdk/go1.22.1/src/os/exec_posix.go:63 +0x78f os.StartProcess() /home/user/sdk/go1.22.1/src/os/exec.go:111 +0x71 os/exec.(*Cmd).Start() /home/user/sdk/go1.22.1/src/os/exec/exec.go:700 +0xac4 github.com/jpillora/overseer.(*parent).fork() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:384 +0xb5e github.com/jpillora/overseer.(*parent).forkLoop() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:356 +0x329 github.com/jpillora/overseer.(*parent).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:65 +0x339 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c Goroutine 80 (running) created at: github.com/jpillora/overseer.(*parent).setupSignalling() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:111 +0x1b0 github.com/jpillora/overseer.(*parent).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:56 +0x1e4 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c ================== ================== WARNING: DATA RACE Read at 0x00c001929ae0 by goroutine 80: sync/atomic.LoadInt32() /home/user/sdk/go1.22.1/src/runtime/race_amd64.s:202 +0xb sync/atomic.LoadUint32() :1 +0x10 os.(*Process).done() /home/user/sdk/go1.22.1/src/os/exec.go:39 +0x230 os.(*Process).signal() /home/user/sdk/go1.22.1/src/os/exec_unix.go:69 +0x233 os.(*Process).Signal() /home/user/sdk/go1.22.1/src/os/exec.go:140 +0x107 github.com/jpillora/overseer.(*parent).sendSignal() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:161 +0xa8 github.com/jpillora/overseer.(*parent).handleSignal() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:144 +0x3a4 github.com/jpillora/overseer.(*parent).setupSignalling.func1() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:113 +0x4b Previous write at 0x00c001929ae0 by main goroutine: os.newProcess() /home/user/sdk/go1.22.1/src/os/exec.go:29 +0x729 os.startProcess() /home/user/sdk/go1.22.1/src/os/exec_posix.go:63 +0x78f os.StartProcess() /home/user/sdk/go1.22.1/src/os/exec.go:111 +0x71 os/exec.(*Cmd).Start() /home/user/sdk/go1.22.1/src/os/exec/exec.go:700 +0xac4 github.com/jpillora/overseer.(*parent).fork() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:384 +0xb5e github.com/jpillora/overseer.(*parent).forkLoop() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:356 +0x329 github.com/jpillora/overseer.(*parent).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:65 +0x339 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c Goroutine 80 (running) created at: github.com/jpillora/overseer.(*parent).setupSignalling() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:111 +0x1b0 github.com/jpillora/overseer.(*parent).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_parent.go:56 +0x1e4 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c ================== ```
2. go-re2 (click here to expand) ``` ================== WARNING: DATA RACE Read at 0x00c000ad4660 by goroutine 1107: github.com/wasilibs/go-re2/internal/alloc.(*mmappedMemory).Reallocate() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/alloc/alloc.go:74 +0x31 github.com/tetratelabs/wazero/internal/wasm.(*MemoryInstance).Grow() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/wasm/memory.go:236 +0x112 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).callWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:309 +0x904 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).CallWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:192 +0x16e github.com/wasilibs/go-re2/internal.(*lazyFunction).callWithStack() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:674 +0x2a9 github.com/wasilibs/go-re2/internal.(*lazyFunction).Call1() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:633 +0x84 github.com/wasilibs/go-re2/internal.malloc() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:508 +0x4a github.com/wasilibs/go-re2/internal.(*libre2ABI).reserve() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:529 +0xce github.com/wasilibs/go-re2/internal.(*libre2ABI).startOperation() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:236 +0xbe github.com/wasilibs/go-re2/internal.(*Regexp).FindAllStringSubmatch() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2.go:439 +0x5f github.com/trufflesecurity/trufflehog/v3/pkg/detectors/github_oauth2.Scanner.FromData() /tmp/t2/pkg/detectors/github_oauth2/github_oauth2.go:45 +0x9d github.com/trufflesecurity/trufflehog/v3/pkg/detectors/github_oauth2.(*Scanner).FromData() :1 +0xc4 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).verificationOverlapWorker() /tmp/t2/pkg/engine/engine.go:886 +0x90a github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers.func1() /tmp/t2/pkg/engine/engine.go:647 +0x190 Previous write at 0x00c000ad4660 by goroutine 2049: github.com/wasilibs/go-re2/internal/alloc.(*mmappedMemory).Reallocate() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/alloc/alloc.go:86 +0xf2 github.com/tetratelabs/wazero/internal/wasm.(*MemoryInstance).Grow() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/wasm/memory.go:236 +0x112 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).callWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:309 +0x904 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).CallWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:192 +0x16e github.com/wasilibs/go-re2/internal.(*lazyFunction).callWithStack() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:674 +0x2a9 github.com/wasilibs/go-re2/internal.(*lazyFunction).Call1() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:633 +0x84 github.com/wasilibs/go-re2/internal.malloc() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:508 +0x4a github.com/wasilibs/go-re2/internal.(*libre2ABI).reserve() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:529 +0xce github.com/wasilibs/go-re2/internal.(*libre2ABI).startOperation() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:236 +0xbe github.com/wasilibs/go-re2/internal.(*Regexp).FindAllStringSubmatch() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2.go:439 +0x5f github.com/trufflesecurity/trufflehog/v3/pkg/detectors/githubapp.Scanner.FromData() /tmp/t2/pkg/detectors/githubapp/githubapp.go:44 +0xc7 github.com/trufflesecurity/trufflehog/v3/pkg/detectors/githubapp.(*Scanner).FromData() :1 +0x93 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).verificationOverlapWorker() /tmp/t2/pkg/engine/engine.go:886 +0x90a github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers.func1() /tmp/t2/pkg/engine/engine.go:647 +0x190 Goroutine 1107 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers() /tmp/t2/pkg/engine/engine.go:643 +0x186 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:603 +0x64 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:572 +0x10c main.runSingleScan() /tmp/t2/main.go:557 +0x687 main.run() /tmp/t2/main.go:426 +0x1113 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c Goroutine 2049 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers() /tmp/t2/pkg/engine/engine.go:643 +0x186 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:603 +0x64 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:572 +0x10c main.runSingleScan() /tmp/t2/main.go:557 +0x687 main.run() /tmp/t2/main.go:426 +0x1113 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c ================== ================== WARNING: DATA RACE Write at 0x00c00137e08c by goroutine 1107: github.com/tetratelabs/wazero/internal/wasm.(*MemoryInstance).Grow() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/wasm/memory.go:245 +0x231 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).callWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:309 +0x904 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).CallWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:192 +0x16e github.com/wasilibs/go-re2/internal.(*lazyFunction).callWithStack() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:674 +0x2a9 github.com/wasilibs/go-re2/internal.(*lazyFunction).Call1() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:633 +0x84 github.com/wasilibs/go-re2/internal.malloc() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:508 +0x4a github.com/wasilibs/go-re2/internal.(*libre2ABI).reserve() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:529 +0xce github.com/wasilibs/go-re2/internal.(*libre2ABI).startOperation() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:236 +0xbe github.com/wasilibs/go-re2/internal.(*Regexp).FindAllStringSubmatch() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2.go:439 +0x5f github.com/trufflesecurity/trufflehog/v3/pkg/detectors/github_oauth2.Scanner.FromData() /tmp/t2/pkg/detectors/github_oauth2/github_oauth2.go:45 +0x9d github.com/trufflesecurity/trufflehog/v3/pkg/detectors/github_oauth2.(*Scanner).FromData() :1 +0xc4 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).verificationOverlapWorker() /tmp/t2/pkg/engine/engine.go:886 +0x90a github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers.func1() /tmp/t2/pkg/engine/engine.go:647 +0x190 Previous write at 0x00c00137e08c by goroutine 2049: github.com/tetratelabs/wazero/internal/wasm.(*MemoryInstance).Grow() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/wasm/memory.go:245 +0x231 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).callWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:309 +0x904 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).CallWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:192 +0x16e github.com/wasilibs/go-re2/internal.(*lazyFunction).callWithStack() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:674 +0x2a9 github.com/wasilibs/go-re2/internal.(*lazyFunction).Call1() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:633 +0x84 github.com/wasilibs/go-re2/internal.malloc() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:508 +0x4a github.com/wasilibs/go-re2/internal.(*libre2ABI).reserve() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:529 +0xce github.com/wasilibs/go-re2/internal.(*libre2ABI).startOperation() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:236 +0xbe github.com/wasilibs/go-re2/internal.(*Regexp).FindAllStringSubmatch() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2.go:439 +0x5f github.com/trufflesecurity/trufflehog/v3/pkg/detectors/githubapp.Scanner.FromData() /tmp/t2/pkg/detectors/githubapp/githubapp.go:44 +0xc7 github.com/trufflesecurity/trufflehog/v3/pkg/detectors/githubapp.(*Scanner).FromData() :1 +0x93 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).verificationOverlapWorker() /tmp/t2/pkg/engine/engine.go:886 +0x90a github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers.func1() /tmp/t2/pkg/engine/engine.go:647 +0x190 Goroutine 1107 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers() /tmp/t2/pkg/engine/engine.go:643 +0x186 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:603 +0x64 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:572 +0x10c main.runSingleScan() /tmp/t2/main.go:557 +0x687 main.run() /tmp/t2/main.go:426 +0x1113 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c Goroutine 2049 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers() /tmp/t2/pkg/engine/engine.go:643 +0x186 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:603 +0x64 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:572 +0x10c main.runSingleScan() /tmp/t2/main.go:557 +0x687 main.run() /tmp/t2/main.go:426 +0x1113 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c ================== ================== WARNING: DATA RACE Read at 0x00c000139f1f by goroutine 1107: encoding/binary.littleEndian.PutUint64() /home/user/sdk/go1.22.1/src/encoding/binary/binary.go:112 +0xe06 github.com/tetratelabs/wazero/internal/engine/wazevo.putLocalMemory() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/module_engine.go:95 +0xd93 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).callWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:320 +0xd44 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).CallWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:192 +0x16e github.com/wasilibs/go-re2/internal.(*lazyFunction).callWithStack() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:674 +0x2a9 github.com/wasilibs/go-re2/internal.(*lazyFunction).Call1() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:633 +0x84 github.com/wasilibs/go-re2/internal.malloc() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:508 +0x4a github.com/wasilibs/go-re2/internal.(*libre2ABI).reserve() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:529 +0xce github.com/wasilibs/go-re2/internal.(*libre2ABI).startOperation() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:236 +0xbe github.com/wasilibs/go-re2/internal.(*Regexp).FindAllStringSubmatch() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2.go:439 +0x5f github.com/trufflesecurity/trufflehog/v3/pkg/detectors/github_oauth2.Scanner.FromData() /tmp/t2/pkg/detectors/github_oauth2/github_oauth2.go:45 +0x9d github.com/trufflesecurity/trufflehog/v3/pkg/detectors/github_oauth2.(*Scanner).FromData() :1 +0xc4 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).verificationOverlapWorker() /tmp/t2/pkg/engine/engine.go:886 +0x90a github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers.func1() /tmp/t2/pkg/engine/engine.go:647 +0x190 Previous write at 0x00c000139f1f by goroutine 2049: encoding/binary.littleEndian.PutUint64() /home/user/sdk/go1.22.1/src/encoding/binary/binary.go:120 +0xf4b github.com/tetratelabs/wazero/internal/engine/wazevo.putLocalMemory() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/module_engine.go:95 +0xd93 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).callWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:320 +0xd44 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).CallWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:192 +0x16e github.com/wasilibs/go-re2/internal.(*lazyFunction).callWithStack() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:674 +0x2a9 github.com/wasilibs/go-re2/internal.(*lazyFunction).Call1() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:633 +0x84 github.com/wasilibs/go-re2/internal.malloc() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:508 +0x4a github.com/wasilibs/go-re2/internal.(*libre2ABI).reserve() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:529 +0xce github.com/wasilibs/go-re2/internal.(*libre2ABI).startOperation() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:236 +0xbe github.com/wasilibs/go-re2/internal.(*Regexp).FindAllStringSubmatch() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2.go:439 +0x5f github.com/trufflesecurity/trufflehog/v3/pkg/detectors/githubapp.Scanner.FromData() /tmp/t2/pkg/detectors/githubapp/githubapp.go:44 +0xc7 github.com/trufflesecurity/trufflehog/v3/pkg/detectors/githubapp.(*Scanner).FromData() :1 +0x93 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).verificationOverlapWorker() /tmp/t2/pkg/engine/engine.go:886 +0x90a github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers.func1() /tmp/t2/pkg/engine/engine.go:647 +0x190 Goroutine 1107 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers() /tmp/t2/pkg/engine/engine.go:643 +0x186 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:603 +0x64 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:572 +0x10c main.runSingleScan() /tmp/t2/main.go:557 +0x687 main.run() /tmp/t2/main.go:426 +0x1113 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c Goroutine 2049 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers() /tmp/t2/pkg/engine/engine.go:643 +0x186 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:603 +0x64 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:572 +0x10c main.runSingleScan() /tmp/t2/main.go:557 +0x687 main.run() /tmp/t2/main.go:426 +0x1113 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c ================== ================== WARNING: DATA RACE Read at 0x00c000139f27 by goroutine 1107: encoding/binary.littleEndian.PutUint64() /home/user/sdk/go1.22.1/src/encoding/binary/binary.go:112 +0xfce github.com/tetratelabs/wazero/internal/engine/wazevo.putLocalMemory() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/module_engine.go:96 +0xf6c github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).callWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:320 +0xd44 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).CallWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:192 +0x16e github.com/wasilibs/go-re2/internal.(*lazyFunction).callWithStack() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:674 +0x2a9 github.com/wasilibs/go-re2/internal.(*lazyFunction).Call1() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:633 +0x84 github.com/wasilibs/go-re2/internal.malloc() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:508 +0x4a github.com/wasilibs/go-re2/internal.(*libre2ABI).reserve() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:529 +0xce github.com/wasilibs/go-re2/internal.(*libre2ABI).startOperation() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:236 +0xbe github.com/wasilibs/go-re2/internal.(*Regexp).FindAllStringSubmatch() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2.go:439 +0x5f github.com/trufflesecurity/trufflehog/v3/pkg/detectors/github_oauth2.Scanner.FromData() /tmp/t2/pkg/detectors/github_oauth2/github_oauth2.go:45 +0x9d github.com/trufflesecurity/trufflehog/v3/pkg/detectors/github_oauth2.(*Scanner).FromData() :1 +0xc4 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).verificationOverlapWorker() /tmp/t2/pkg/engine/engine.go:886 +0x90a github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers.func1() /tmp/t2/pkg/engine/engine.go:647 +0x190 Previous write at 0x00c000139f27 by goroutine 2049: encoding/binary.littleEndian.PutUint64() /home/user/sdk/go1.22.1/src/encoding/binary/binary.go:120 +0x1112 github.com/tetratelabs/wazero/internal/engine/wazevo.putLocalMemory() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/module_engine.go:96 +0xf6c github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).callWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:320 +0xd44 github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).CallWithStack() /home/user/go/pkg/mod/github.com/tetratelabs/wazero@v1.7.1/internal/engine/wazevo/call_engine.go:192 +0x16e github.com/wasilibs/go-re2/internal.(*lazyFunction).callWithStack() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:674 +0x2a9 github.com/wasilibs/go-re2/internal.(*lazyFunction).Call1() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:633 +0x84 github.com/wasilibs/go-re2/internal.malloc() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:508 +0x4a github.com/wasilibs/go-re2/internal.(*libre2ABI).reserve() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:529 +0xce github.com/wasilibs/go-re2/internal.(*libre2ABI).startOperation() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2_wazero.go:236 +0xbe github.com/wasilibs/go-re2/internal.(*Regexp).FindAllStringSubmatch() /home/user/go/pkg/mod/github.com/wasilibs/go-re2@v1.5.3/internal/re2.go:439 +0x5f github.com/trufflesecurity/trufflehog/v3/pkg/detectors/githubapp.Scanner.FromData() /tmp/t2/pkg/detectors/githubapp/githubapp.go:44 +0xc7 github.com/trufflesecurity/trufflehog/v3/pkg/detectors/githubapp.(*Scanner).FromData() :1 +0x93 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).verificationOverlapWorker() /tmp/t2/pkg/engine/engine.go:886 +0x90a github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers.func1() /tmp/t2/pkg/engine/engine.go:647 +0x190 Goroutine 1107 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers() /tmp/t2/pkg/engine/engine.go:643 +0x186 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:603 +0x64 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:572 +0x10c main.runSingleScan() /tmp/t2/main.go:557 +0x687 main.run() /tmp/t2/main.go:426 +0x1113 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c Goroutine 2049 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers() /tmp/t2/pkg/engine/engine.go:643 +0x186 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:603 +0x64 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:572 +0x10c main.runSingleScan() /tmp/t2/main.go:557 +0x687 main.run() /tmp/t2/main.go:426 +0x1113 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c ================== ```
3. decoder (click here to expand) ``` ================== WARNING: DATA RACE Read at 0x00c00c636158 by goroutine 2045: runtime.slicebytetostring() /home/user/sdk/go1.22.1/src/runtime/string.go:81 +0x0 github.com/trufflesecurity/trufflehog/v3/pkg/detectors/circleci.Scanner.FromData() /tmp/t2/pkg/detectors/circleci/circleci.go:31 +0x85 github.com/trufflesecurity/trufflehog/v3/pkg/detectors/circleci.(*Scanner).FromData() :1 +0x93 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).verificationOverlapWorker() /tmp/t2/pkg/engine/engine.go:886 +0x90a github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers.func1() /tmp/t2/pkg/engine/engine.go:647 +0x190 Previous write at 0x00c00c636158 by goroutine 90: runtime.slicecopy() /home/user/sdk/go1.22.1/src/runtime/slice.go:325 +0x0 github.com/trufflesecurity/trufflehog/v3/pkg/decoders.decodeEscaped() /tmp/t2/pkg/decoders/escaped_unicode.go:123 +0x3d8 github.com/trufflesecurity/trufflehog/v3/pkg/decoders.(*EscapedUnicode).FromChunk() /tmp/t2/pkg/decoders/escaped_unicode.go:39 +0x217 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).scannerWorker() /tmp/t2/pkg/engine/engine.go:735 +0x6e8 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startScannerWorkers.func1() /tmp/t2/pkg/engine/engine.go:618 +0x190 Goroutine 2045 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startVerificationOverlapWorkers() /tmp/t2/pkg/engine/engine.go:643 +0x186 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:603 +0x64 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:572 +0x10c main.runSingleScan() /tmp/t2/main.go:557 +0x687 main.run() /tmp/t2/main.go:426 +0x1113 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c Goroutine 90 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startScannerWorkers() /tmp/t2/pkg/engine/engine.go:614 +0x186 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:596 +0x3a github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:572 +0x10c main.runSingleScan() /tmp/t2/main.go:557 +0x687 main.run() /tmp/t2/main.go:426 +0x1113 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c ================== ```
4. Detector(s) (click here to expand) ``` ================== WARNING: DATA RACE Write at 0x00000b058d90 by goroutine 756: github.com/trufflesecurity/trufflehog/v3/pkg/detectors/ldap.verifyLDAP() /tmp/t2/pkg/detectors/ldap/ldap.go:130 +0x66 github.com/trufflesecurity/trufflehog/v3/pkg/detectors/ldap.Scanner.FromData() /tmp/t2/pkg/detectors/ldap/ldap.go:71 +0xb2b github.com/trufflesecurity/trufflehog/v3/pkg/detectors/ldap.(*Scanner).FromData() :1 +0x93 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).detectChunk() /tmp/t2/pkg/engine/engine.go:820 +0x422 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).detectorWorker() /tmp/t2/pkg/engine/engine.go:794 +0x16f github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startDetectorWorkers.func1() /tmp/t2/pkg/engine/engine.go:593 +0x190 Previous read at 0x00000b058d90 by goroutine 336: github.com/go-ldap/ldap/v3.DialURL() /home/user/go/pkg/mod/github.com/go-ldap/ldap/v3@v3.4.8/conn.go:240 +0x23e github.com/trufflesecurity/trufflehog/v3/pkg/detectors/ldap.verifyLDAP() /tmp/t2/pkg/detectors/ldap/ldap.go:137 +0xe4 github.com/trufflesecurity/trufflehog/v3/pkg/detectors/ldap.Scanner.FromData() /tmp/t2/pkg/detectors/ldap/ldap.go:71 +0xb2b github.com/trufflesecurity/trufflehog/v3/pkg/detectors/ldap.(*Scanner).FromData() :1 +0x93 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).detectChunk() /tmp/t2/pkg/engine/engine.go:820 +0x422 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).detectorWorker() /tmp/t2/pkg/engine/engine.go:794 +0x16f github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startDetectorWorkers.func1() /tmp/t2/pkg/engine/engine.go:593 +0x190 Goroutine 756 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startDetectorWorkers() /tmp/t2/pkg/engine/engine.go:589 +0x191 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:563 +0x4e github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:536 +0x10c main.runSingleScan() /tmp/t2/main.go:556 +0x687 main.run() /tmp/t2/main.go:425 +0x10d3 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c Goroutine 336 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startDetectorWorkers() /tmp/t2/pkg/engine/engine.go:589 +0x191 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:563 +0x4e github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:536 +0x10c main.runSingleScan() /tmp/t2/main.go:556 +0x687 main.run() /tmp/t2/main.go:425 +0x10d3 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c ================== ================== WARNING: DATA RACE Write at 0x00000b058d90 by goroutine 756: github.com/trufflesecurity/trufflehog/v3/pkg/detectors/ldap.verifyLDAP() /tmp/t2/pkg/detectors/ldap/ldap.go:130 +0x66 github.com/trufflesecurity/trufflehog/v3/pkg/detectors/ldap.Scanner.FromData() /tmp/t2/pkg/detectors/ldap/ldap.go:71 +0xb2b github.com/trufflesecurity/trufflehog/v3/pkg/detectors/ldap.(*Scanner).FromData() :1 +0x93 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).detectChunk() /tmp/t2/pkg/engine/engine.go:820 +0x422 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).detectorWorker() /tmp/t2/pkg/engine/engine.go:794 +0x16f github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startDetectorWorkers.func1() /tmp/t2/pkg/engine/engine.go:593 +0x190 Previous read at 0x00000b058d90 by goroutine 336: github.com/go-ldap/ldap/v3.DialURL() /home/user/go/pkg/mod/github.com/go-ldap/ldap/v3@v3.4.8/conn.go:240 +0x23e github.com/trufflesecurity/trufflehog/v3/pkg/detectors/ldap.verifyLDAP() /tmp/t2/pkg/detectors/ldap/ldap.go:157 +0x6e5 github.com/trufflesecurity/trufflehog/v3/pkg/detectors/ldap.Scanner.FromData() /tmp/t2/pkg/detectors/ldap/ldap.go:71 +0xb2b github.com/trufflesecurity/trufflehog/v3/pkg/detectors/ldap.(*Scanner).FromData() :1 +0x93 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).detectChunk() /tmp/t2/pkg/engine/engine.go:820 +0x422 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).detectorWorker() /tmp/t2/pkg/engine/engine.go:794 +0x16f github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startDetectorWorkers.func1() /tmp/t2/pkg/engine/engine.go:593 +0x190 Goroutine 756 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startDetectorWorkers() /tmp/t2/pkg/engine/engine.go:589 +0x191 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:563 +0x4e github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:536 +0x10c main.runSingleScan() /tmp/t2/main.go:556 +0x687 main.run() /tmp/t2/main.go:425 +0x10d3 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c Goroutine 336 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startDetectorWorkers() /tmp/t2/pkg/engine/engine.go:589 +0x191 github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startWorkers() /tmp/t2/pkg/engine/engine.go:563 +0x4e github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).Start() /tmp/t2/pkg/engine/engine.go:536 +0x10c main.runSingleScan() /tmp/t2/main.go:556 +0x687 main.run() /tmp/t2/main.go:425 +0x10d3 github.com/jpillora/overseer.(*child).run() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/proc_child.go:77 +0x5b5 github.com/jpillora/overseer.runErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:164 +0x278 github.com/jpillora/overseer.RunErr() /home/user/go/pkg/mod/github.com/trufflesecurity/overseer@v1.2.7/overseer.go:94 +0x5a4 main.main() /tmp/t2/main.go:294 +0x52c ================== ```
GitHub setScanOptions (click here to expand) ``` ================== WARNING: DATA RACE Write at 0x00c0058cf208 by goroutine 22192: github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).setScanOptions() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:104 +0xe4 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).cloneAndScanRepo() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:841 +0x7c4 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:751 +0x42b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x91 Previous read at 0x00c0058cf208 by goroutine 17836: github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).ScanStaged() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:850 +0x219 github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).ScanRepo() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:979 +0x1f6 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).cloneAndScanRepo() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:844 +0x865 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:751 +0x42b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x91 Goroutine 22192 (running) created at: golang.org/x/sync/errgroup.(*Group).Go() /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0x124 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:723 +0x508 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).Chunks() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:386 +0x44a github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).runWithoutUnits() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:311 +0x39d github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).run() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:286 +0xa44 github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).Run.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:162 +0x41a Goroutine 17836 (running) created at: golang.org/x/sync/errgroup.(*Group).Go() /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0x124 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:723 +0x508 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).Chunks() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:386 +0x44a github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).runWithoutUnits() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:311 +0x39d github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).run() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:286 +0xa44 github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).Run.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:162 +0x41a ================== ================== WARNING: DATA RACE Write at 0x00c0058cf218 by goroutine 22192: github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).setScanOptions() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:105 +0x147 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).cloneAndScanRepo() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:841 +0x7c4 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:751 +0x42b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x91 Previous read at 0x00c0058cf218 by goroutine 17836: github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).ScanStaged() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:853 +0x390 github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).ScanRepo() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:979 +0x1f6 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).cloneAndScanRepo() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:844 +0x865 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:751 +0x42b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x91 Goroutine 22192 (running) created at: golang.org/x/sync/errgroup.(*Group).Go() /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0x124 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:723 +0x508 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).Chunks() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:386 +0x44a github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).runWithoutUnits() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:311 +0x39d github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).run() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:286 +0xa44 github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).Run.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:162 +0x41a Goroutine 17836 (running) created at: golang.org/x/sync/errgroup.(*Group).Go() /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0x124 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:723 +0x508 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).Chunks() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:386 +0x44a github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).runWithoutUnits() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:311 +0x39d github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).run() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:286 +0xa44 github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).Run.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:162 +0x41a ================== ```
Handlers/BufferedFileReader (click here to expand) ``` ================== WARNING: DATA RACE Write at 0x00c0044321d0 by goroutine 162778: github.com/trufflesecurity/trufflehog/v3/pkg/buffers/buffer.(*readCloser).Close() /home/user/dev/github.com/trufflesecurity/thog2/pkg/buffers/buffer/buffer.go:93 +0x6b github.com/trufflesecurity/trufflehog/v3/pkg/readers.(*BufferedFileReader).Close() /home/user/dev/github.com/trufflesecurity/thog2/pkg/readers/bufferedfilereader.go:50 +0x42 github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).openArchive.(*archiveHandler).extractorHandler.func1.deferwrap2() /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:196 +0x33 runtime.deferreturn() /home/user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.2.linux-amd64/src/runtime/panic.go:602 +0x5d github.com/mholt/archiver/v4.Zip.Extract() /home/user/go/pkg/mod/github.com/mholt/archiver/v4@v4.0.0-alpha.8.0.20240408183022-de08bfa4c558/zip.go:226 +0x58a github.com/mholt/archiver/v4.(*Zip).Extract() :1 +0xf7 github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).openArchive() /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:116 +0x5c5 github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).HandleFile.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:66 +0x29e Previous read at 0x00c0044321d0 by goroutine 163765: github.com/trufflesecurity/trufflehog/v3/pkg/buffers/buffer.(*readCloser).Read() /home/user/dev/github.com/trufflesecurity/thog2/pkg/buffers/buffer/buffer.go:102 +0x44 github.com/trufflesecurity/trufflehog/v3/pkg/readers.(*BufferedFileReader).Read() /home/user/dev/github.com/trufflesecurity/thog2/pkg/readers/bufferedfilereader.go:56 +0x69 bufio.(*Reader).Read() /home/user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.2.linux-amd64/src/bufio/bufio.go:227 +0x2ab bufio.(*Reader).fill() /home/user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.2.linux-amd64/src/bufio/bufio.go:110 +0x2af bufio.(*Reader).Peek() /home/user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.2.linux-amd64/src/bufio/bufio.go:148 +0xc6 github.com/trufflesecurity/trufflehog/v3/pkg/sources.readInChunks.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/chunker.go:140 +0x21d Goroutine 162778 (running) created at: github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).HandleFile() /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:53 +0x1a4 github.com/trufflesecurity/trufflehog/v3/pkg/handlers.HandleFile() /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/handlers.go:197 +0x68f github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).handleBinary() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:1305 +0x1014 github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).ScanCommits() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:694 +0x163d github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).ScanRepo() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:975 +0x17e github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).cloneAndScanRepo() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:844 +0x865 github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:751 +0x42b golang.org/x/sync/errgroup.(*Group).Go.func1() /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x91 Goroutine 163765 (finished) created at: github.com/trufflesecurity/trufflehog/v3/pkg/sources.readInChunks() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/chunker.go:131 +0x332 github.com/trufflesecurity/trufflehog/v3/pkg/sources.NewChunkReader.createReaderFn.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/chunker.go:122 +0x52 github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*defaultHandler).handleNonArchiveContent() /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/default.go:98 +0x352 github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).openArchive() /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:92 +0x15b github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).openArchive.(*archiveHandler).extractorHandler.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:201 +0xc51 github.com/mholt/archiver/v4.Zip.Extract() /home/user/go/pkg/mod/github.com/mholt/archiver/v4@v4.0.0-alpha.8.0.20240408183022-de08bfa4c558/zip.go:226 +0x58a github.com/mholt/archiver/v4.(*Zip).Extract() :1 +0xf7 github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).openArchive() /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:116 +0x5c5 github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).HandleFile.func1() /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:66 +0x29e ================== ```
rgmz commented 4 months ago

FWIW, I was able to eliminate all races by doing the following:

  1. Deleted all "verificationOverlap" related code in engine.go
  2. Removed go-re2 in favour of the stdlib's regex package
  3. Cloned data passed to the EscapedUnicode decoder

    diff --git a/pkg/decoders/escaped_unicode.go b/pkg/decoders/escaped_unicode.go
    --- a/pkg/decoders/escaped_unicode.go
    +++ b/pkg/decoders/escaped_unicode.go
    @@ -1,6 +1,7 @@
     package decoders
    
     import (
    +       "bytes"
            "regexp"
            "strconv"
            "unicode/utf8"
    @@ -30,19 +31,29 @@ func (d *EscapedUnicode) FromChunk(chunk *sources.Chunk) *DecodableChunk {
            }
    
            matched := false
    -       if codePointPat.Match(chunk.Data) {
    +       chunkData := bytes.Clone(chunk.Data)
    +       if codePointPat.Match(chunkData) {
                    matched = true
    -               chunk.Data = decodeCodePoint(chunk.Data)
    +               chunkData = decodeCodePoint(chunkData)
            }
            if escapePat.Match(chunk.Data) {
                    matched = true
    -               chunk.Data = decodeEscaped(chunk.Data)
    +               chunkData = decodeEscaped(chunkData)
            }
    
            if matched {
                    decodableChunk := &DecodableChunk{
                            DecoderType: detectorspb.DecoderType_ESCAPED_UNICODE,
    -                       Chunk:       chunk,
    +                       Chunk: &sources.Chunk{
    +                               Data:           chunkData,
    +                               SourceName:     chunk.SourceName,
    +                               SourceID:       chunk.SourceID,
    +                               JobID:          chunk.JobID,
    +                               SecretID:       chunk.SecretID,
    +                               SourceMetadata: chunk.SourceMetadata,
    +                               SourceType:     chunk.SourceType,
    +                               Verify:         chunk.Verify,
    +                       },
                    }
                    return decodableChunk
            } else {
  4. Updating caflou and ldap to avoid touching global variables inside the FromData() flow
    
    diff --git a/pkg/detectors/ldap/ldap.go b/pkg/detectors/ldap/ldap.go
    --- a/pkg/detectors/ldap/ldap.go
    +++ b/pkg/detectors/ldap/ldap.go
    @@ -6,23 +6,27 @@ import (
    +func init() {
    +       ldap.DefaultTimeout = 5 * time.Second
    +}
    +
    var (
        // Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
        uriPat = regexp.MustCompile(`\b(?i)ldaps?://[\S]+\b`)
    @@ -127,8 +131,6 @@ func isErrDeterminate(err error) bool {
    func verifyLDAP(username, password string, ldapURL *url.URL) error {
        // Tests with non-TLS, TLS, and STARTTLS

-type Scanner struct{} +type Scanner struct {

Obviously, #2 is probably the most ??? issue.

rgmz commented 4 months ago

Another one:

==================
WARNING: DATA RACE
Write at 0x00c0058cf208 by goroutine 22192:
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).setScanOptions()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:104 +0xe4
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).cloneAndScanRepo()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:841 +0x7c4
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:751 +0x42b
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x91

Previous read at 0x00c0058cf208 by goroutine 17836:
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).ScanStaged()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:850 +0x219
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).ScanRepo()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:979 +0x1f6
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).cloneAndScanRepo()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:844 +0x865
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:751 +0x42b
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x91

Goroutine 22192 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0x124
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:723 +0x508
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).Chunks()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:386 +0x44a
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).runWithoutUnits()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:311 +0x39d
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).run()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:286 +0xa44
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).Run.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:162 +0x41a

Goroutine 17836 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0x124
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:723 +0x508
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).Chunks()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:386 +0x44a
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).runWithoutUnits()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:311 +0x39d
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).run()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:286 +0xa44
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).Run.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:162 +0x41a
==================
==================
WARNING: DATA RACE
Write at 0x00c0058cf218 by goroutine 22192:
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).setScanOptions()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:105 +0x147
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).cloneAndScanRepo()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:841 +0x7c4
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:751 +0x42b
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x91

Previous read at 0x00c0058cf218 by goroutine 17836:
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).ScanStaged()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:853 +0x390
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).ScanRepo()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:979 +0x1f6
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).cloneAndScanRepo()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:844 +0x865
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:751 +0x42b
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x91

Goroutine 22192 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0x124
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:723 +0x508
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).Chunks()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:386 +0x44a
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).runWithoutUnits()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:311 +0x39d
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).run()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:286 +0xa44
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).Run.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:162 +0x41a

Goroutine 17836 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0x124
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:723 +0x508
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).Chunks()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:386 +0x44a
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).runWithoutUnits()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:311 +0x39d
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).run()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:286 +0xa44
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.(*SourceManager).Run.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/source_manager.go:162 +0x41a
==================
rgmz commented 4 months ago

Another one. This seems to occur when there's a nested archive that takes a while to read.

==================
WARNING: DATA RACE
Write at 0x00c0044321d0 by goroutine 162778:
  github.com/trufflesecurity/trufflehog/v3/pkg/buffers/buffer.(*readCloser).Close()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/buffers/buffer/buffer.go:93 +0x6b
  github.com/trufflesecurity/trufflehog/v3/pkg/readers.(*BufferedFileReader).Close()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/readers/bufferedfilereader.go:50 +0x42
  github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).openArchive.(*archiveHandler).extractorHandler.func1.deferwrap2()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:196 +0x33
  runtime.deferreturn()
      /home/user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.2.linux-amd64/src/runtime/panic.go:602 +0x5d
  github.com/mholt/archiver/v4.Zip.Extract()
      /home/user/go/pkg/mod/github.com/mholt/archiver/v4@v4.0.0-alpha.8.0.20240408183022-de08bfa4c558/zip.go:226 +0x58a
  github.com/mholt/archiver/v4.(*Zip).Extract()
      <autogenerated>:1 +0xf7
  github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).openArchive()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:116 +0x5c5
  github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).HandleFile.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:66 +0x29e

Previous read at 0x00c0044321d0 by goroutine 163765:
  github.com/trufflesecurity/trufflehog/v3/pkg/buffers/buffer.(*readCloser).Read()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/buffers/buffer/buffer.go:102 +0x44
  github.com/trufflesecurity/trufflehog/v3/pkg/readers.(*BufferedFileReader).Read()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/readers/bufferedfilereader.go:56 +0x69
  bufio.(*Reader).Read()
      /home/user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.2.linux-amd64/src/bufio/bufio.go:227 +0x2ab
  bufio.(*Reader).fill()
      /home/user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.2.linux-amd64/src/bufio/bufio.go:110 +0x2af
  bufio.(*Reader).Peek()
      /home/user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.2.linux-amd64/src/bufio/bufio.go:148 +0xc6
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.readInChunks.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/chunker.go:140 +0x21d

Goroutine 162778 (running) created at:
  github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).HandleFile()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:53 +0x1a4
  github.com/trufflesecurity/trufflehog/v3/pkg/handlers.HandleFile()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/handlers.go:197 +0x68f
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).handleBinary()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:1305 +0x1014
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).ScanCommits()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:694 +0x163d
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/git.(*Git).ScanRepo()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/git/git.go:975 +0x17e
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).cloneAndScanRepo()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:844 +0x865
  github.com/trufflesecurity/trufflehog/v3/pkg/sources/github.(*Source).scan.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/github/github.go:751 +0x42b
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/user/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x91

Goroutine 163765 (finished) created at:
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.readInChunks()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/chunker.go:131 +0x332
  github.com/trufflesecurity/trufflehog/v3/pkg/sources.NewChunkReader.createReaderFn.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/sources/chunker.go:122 +0x52
  github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*defaultHandler).handleNonArchiveContent()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/default.go:98 +0x352
  github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).openArchive()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:92 +0x15b
  github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).openArchive.(*archiveHandler).extractorHandler.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:201 +0xc51
  github.com/mholt/archiver/v4.Zip.Extract()
      /home/user/go/pkg/mod/github.com/mholt/archiver/v4@v4.0.0-alpha.8.0.20240408183022-de08bfa4c558/zip.go:226 +0x58a
  github.com/mholt/archiver/v4.(*Zip).Extract()
      <autogenerated>:1 +0xf7
  github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).openArchive()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:116 +0x5c5
  github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).HandleFile.func1()
      /home/user/dev/github.com/trufflesecurity/thog2/pkg/handlers/archive.go:66 +0x29e
==================
JoelLau commented 4 weeks ago

hi @rgmz , i was wondering if you're encountered this error while running go build -race on this repo before

# github.com/trufflesecurity/trufflehog/v3
ld: warning: '/private/var/folders/zp/f2pjs1x57rg0hqy5cf2ffvqr0000gq/T/go-link-1268426122/000060.o' has malformed LC_DYSYMTAB, expected 98 undefined symbols to start at index 1626, found 95 undefined symbols starting at index 1626
rgmz commented 4 weeks ago

No. That seems to be a warning but nothing serious, per https://github.com/golang/go/issues/61229.