Scan Error: Out of Memory

[KaSiemund]

When trying to scan my current git repo, I get the following error

Reproduction not possible, because coding is confidential.

I expected the scan to run without issues, as it also runs in other repos in which I develop.

OS: Windows 10 Enterprise From my perspective the repo is relatively large and there are also jupyter notebooks included.

@jmatias

Full Error Message: C:\Users\USER\PycharmProjects\Project>talisman --scan Talisman Fetch Blobs: 254 / 254 <---------------------------------------------------------------------------> 100.00%

d8888b. db db d8b db d8b db d888888b d8b db d888b .d8888. .o88b. .d8b. d8b db 88 8D 88 88 888o 88 888o 8888' 888o 88 88' Y8b 88' YP d8P Y8 d8' 8b 888o 88 88oobY' 88 88 88V8o 88 88V8o 88 88 88V8o 88 888bo. 8P 88ooo88 88V8o 88 888b 88 88 88 V8o88 88 V8o88 88 88 V8o88 88 oooY8b. 8b 88~~~88 88 V8o88 88 88. 88b d88 88 V888 88 V888 .88. 88 V888 88. ~8~ db 8D Y8b d8 88 88 88 V888 db db 88 YD ~Y8888P' VP V8P VP V8P Y888888P VP V8P Y888P8888Y' `Y88P' YP YP VP V8P VP VP

Talisman Scan: 1361 / 2133 <---------------------------------------------------↘.............................> 63.81% runtime: out of memory: cannot allocate 104857600-byte block (1192984576 in use) fatal error: out of memory

goroutine 949 [running]: runtime.throw({0xfe3851, 0xd}) /Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/panic.go:992 +0x64 fp=0x22c9ae6c sp=0x22c9ae58 pc=0xcf6f94 runtime.(mcache).allocLarge(0x1cd0c18, 0x62b8000, 0x1) /Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/mcache.go:215 +0x1e9 fp=0x22c9ae94 sp=0x22c9ae6c pc=0xcd4589 runtime.mallocgc(0x62b8000, 0x0, 0x0) /Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/malloc.go:1096 +0x446 fp=0x22c9aedc sp=0x22c9ae94 pc=0xccbc56 runtime.rawbyteslice(0x62b627a) /Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/string.go:286 +0xdb fp=0x22c9aef8 sp=0x22c9aedc pc=0xd0fe3b runtime.stringtoslicebyte(0x0, {0x27f9e000, 0x62b627a}) /Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/string.go:178 +0x4c fp=0x22c9af18 sp=0x22c9aef8 pc=0xd0f8dc talisman/detector/filecontent.(FileContentDetector).Test.func1({{0x12e8c4dd, 0x2c}, {0x12e8c4f2, 0x17}, {0x124061d0, 0x1, 0x1}, {0x14fe0000, 0x62b627a, 0x62b627a}}) /Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:122 +0x210 fp=0x22c9afc4 sp=0x22c9af18 pc=0xf5d4f0 talisman/detector/filecontent.(FileContentDetector).Test.func3() /Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:132 +0x2e fp=0x22c9aff0 sp=0x22c9afc4 pc=0xf5d2be runtime.goexit() /Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/asm_386.s:1326 +0x1 fp=0x22c9aff4 sp=0x22c9aff0 pc=0xd23791 created by talisman/detector/filecontent.(FileContentDetector).Test /Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:109 +0x717

goroutine 1 [runnable]: talisman/detector/filecontent.processContent({{0x12d680ff, 0xc}, {0x12d680ff, 0xc}, 0x1, {0x5d667800, 0x668, 0x700}, 0x3}, 0x0, ...) /Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:170 +0xa6 talisman/detector/filecontent.(FileContentDetector).Test(0x125ae240, {{0x12bc2a0, 0x124a93e0}, {0x12bc644, 0x14efcdc}, 0x124cc0a0}, {0x12ba2000, 0x2c7, 0x400}, 0x124cc0a0, ...) /Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:154 +0x527 talisman/detector.(Chain).Test(0x125ae210, {0x12ba2000, 0x2c7, 0x400}, 0x124cc0a0, 0x12cec6c0) /Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/chain.go:62 +0x46f main.(ScannerCmd).Run(0x125ae030, 0x124cc0a0) /Users/javiermatias-cabrera/workspace/external-repos/talisman/cmd/scanner_cmd.go:32 +0x102 main.run({0x0, {0x12bbea0, 0x14efcdc}}) /Users/javiermatias-cabrera/workspace/external-repos/talisman/cmd/talisman.go:152 +0x753 main.main() /Users/javiermatias-cabrera/workspace/external-repos/talisman/cmd/talisman.go:124 +0x2c1 ... goroutine 1396 [semacquire]: sync.runtime_Semacquire(0x128d87f8) /Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/sema.go:56 +0x36 sync.(WaitGroup).Wait(0x128d87f0) /Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/sync/waitgroup.go:136 +0x87 talisman/detector/filecontent.(FileContentDetector).Test.func2() /Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:135 +0x2e created by talisman/detector/filecontent.(FileContentDetector).Test /Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:134 +0x290