Valid GZIP file: `gzip: invalid header`

rgmz commented 3 weeks ago

Please review the Community Note before submitting

TruffleHog Version

3.78.0

Trace Output

$ ./trufflehog/trufflehog filesystem Conifer_Desktop.spm
🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2024-06-06T18:54:16-04:00       info-0  trufflehog      running source  {"source_manager_worker_id": "R0FWD", "with_units": true}
2024-06-06T18:54:16-04:00       error   trufflehog      error unarchiving chunk.        {"source_manager_worker_id": "R0FWD", "unit": "Conifer_Desktop.spm", "unit_kind": "unit", "timeout": 30, "error": "error creating custom reader: error creating random access reader: error creating BufferedFileReader: error writing to buffered file writer: gzip: invalid header"}
2024-06-06T18:54:16-04:00       info-0  trufflehog      finished scanning       {"chunks": 0, "bytes": 0, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "21.628122ms", "trufflehog_version": "dev"}

Expected Behavior

The file should be detected as GZIP, extracted, and scanned.

$ file -i Conifer_Desktop.spm
Conifer_Desktop.spm: application/gzip; charset=binary
$ gunzip -f Conifer_Desktop.spm -S .spm

gzip: Conifer_Desktop.spm: decompression OK, trailing garbage ignored

Actual Behavior

The file is not scanned due to an invalid header error.

Steps to Reproduce

Download https://github.com/akuang/self-driving-car-sim/blob/4b1f739ebda9ed4920fe895ee3677bd4ccb79218/Assets/Standard%20Assets/Environment/SpeedTree/Conifer/Conifer_Desktop.spm
Run TruffleHog against the file

Environment

N/A

Additional Context

This may or may not be related to #2933.

References

N/A

rgmz commented 3 weeks ago

Another one: https://github.com/RedHatDemos/labstation/blob/926a2441d5c61c7bbab259c618c9793efea8ed15/homedir/.cache/mozilla/firefox/yxzu1t1m.default/cache2/entries/833D3306D1E0B3A394046E97977230E84FD76477

CLI

$ file -i 833D3306D1E0B3A394046E97977230E84FD76477
833D3306D1E0B3A394046E97977230E84FD76477: application/gzip; charset=binary
$ mv 833D3306D1E0B3A394046E97977230E84FD76477{,.gz}
$ gunzip -S "gz" 833D3306D1E0B3A394046E97977230E84FD76477.gz

gzip: 833D3306D1E0B3A394046E97977230E84FD76477.gz: decompression OK, trailing garbage ignored

TruffleHog

$ ./trufflehog/trufflehog filesystem /tmp/833D3306D1E0B3A394046E97977230E84FD76477.gz
🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2024-06-07T10:10:28-04:00       info-0  trufflehog      running source  {"source_manager_worker_id": "kDgXL", "with_units": true}
2024-06-07T10:10:28-04:00       error   trufflehog      error unarchiving chunk.        {"source_manager_worker_id": "kDgXL", "unit": "/tmp/833D3306D1E0B3A394046E97977230E84FD76477.gz", "unit_kind": "unit", "timeout": 30, "error": "error creating custom reader: error creating random access reader: error creating BufferedFileReader: error writing to buffered file writer: gzip: invalid header"}
2024-06-07T10:10:28-04:00       info-0  trufflehog      finished scanning       {"chunks": 0, "bytes": 0, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "6.039105ms", "trufflehog_version": "dev"}

rgmz commented 3 weeks ago

The stack traces for both seem to be the same cause. It's unclear whether this is an issue with the mholt/archiver library, or how TruffleHog reads the data.

https://github.com/akuang/self-driving-car-sim/blob/4b1f739ebda9ed4920fe895ee3677bd4ccb79218/Assets/Standard%20Assets/Environment/SpeedTree/Conifer/Conifer_Desktop.spm

2024-06-07T12:51:03-04:00       info-0  trufflehog      archive.Decompressor: error creating custom reader      {"source_manager_worker_id": "DAxip", "unit": "Conifer_Desktop.spm", "unit_kind": "unit", "timeout": 30}
panic: error creating random access reader: error creating BufferedFileReader: error writing to buffered file writer: gzip: invalid header

goroutine 2147 [running]:
github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).openArchive(0xc001dee038, {0x4c67660, 0xc0017e2570}, 0x0, {{0x4c43120, 0xc000014910}, {0x3f6085a, 0x3}, 0xc000b460d8, 0x1}, ...)
        /tmp/trufflehog/pkg/handlers/archive.go:109 +0xbec
github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).HandleFile.func1()
        /tmp/trufflehog/pkg/handlers/archive.go:64 +0x17d
created by github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).HandleFile in goroutine 2146
        /tmp/trufflehog/pkg/handlers/archive.go:51 +0xfb

https://github.com/RedHatDemos/labstation/blob/926a2441d5c61c7bbab259c618c9793efea8ed15/homedir/.cache/mozilla/firefox/yxzu1t1m.default/cache2/entries/833D3306D1E0B3A394046E97977230E84FD76477

2024-06-07T12:51:54-04:00       info-0  trufflehog      archive.Decompressor: error creating custom reader      {"source_manager_worker_id": "p9bYg", "unit": "833D3306D1E0B3A394046E97977230E84FD76477", "unit_kind": "unit", "timeout": 30}
panic: error creating random access reader: error creating BufferedFileReader: error writing to buffered file writer: gzip: invalid header

goroutine 94 [running]:
github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).openArchive(0xc000704720, {0x4c67660, 0xc0024fc300}, 0x0, {{0x4c43120, 0xc002ddf1e0}, {0x3f6085a, 0x3}, 0xc002de0918, 0x1}, ...)
        /tmp/trufflehog/pkg/handlers/archive.go:109 +0xbec
github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).HandleFile.func1()
        /tmp/trufflehog/pkg/handlers/archive.go:64 +0x17d
created by github.com/trufflesecurity/trufflehog/v3/pkg/handlers.(*archiveHandler).HandleFile in goroutine 93
        /tmp/trufflehog/pkg/handlers/archive.go:51 +0xfb

rgmz commented 2 weeks ago

I've created an issue in the upstream library: https://github.com/klauspost/compress/issues/972

trufflesecurity / trufflehog