LinkFinder high RAM usage causes system crash

[frost19k]

Describe the bug Line 1251 which calls LinkFinder on entries in js/js_livelinks.txt takes up all system memory, including swap, crashing the system.

To Reproduce

docker run -it --rm --cpus=12 --memory=24g \
-v ${PWD}/Recon:/root/Tools/reconftw/Recon/ \
-v ${PWD}/reconftw.cfg:/root/Tools/reconftw/reconftw.cfg \
six2dez/reconftw:main -d target.com -c jschecks 

Expected behavior Function jschecks should complete successfully without crashing the system.

Screenshots

Desktop (please complete the following information):

• Architecture: x86_64
  • OS: Manjaro
  • Version: Rolling (currently 21.2.5)

Additional context

┌──(root㉿9bcb8dc1b54e)-[/root/Tools/reconftw/Recon/<target.com>]
└─# wc -l js/js_livelinks.txt 
7967 js/js_livelinks.txt

As can be seen in the screenshot, 32 gigs of RAM & 16 gigs of swap are (almost) fully consumed. I killed the process before it got to 100% because then the system would become unresponsive - requiring a hard reboot.

I've tried limiting interlace to 2 threads - this just slows down the inevitable crash.

Limiting the Docker container's threads & memory has no effect.

Seems like run-mailcap is the culprit - will look into this further.

In the meantime, I'd really appreciate advice on further troubleshooting. I feel like I'm outta ideas.

[six2dez]

This seems to be something really specific from linkfinder tool, it would be probably better to report into LinkFinder repository, anyway I will try to reproduce this and keep you updated.

Thanks!

[frost19k]

So, I was making a mistake with my Docker command. There's another parameter --memory-swap which is set to the same value as --memory if not explicitly defined. Essentially, I was allotting Docker 48GB of memory + swap - which is more than I have.

When I instead do --memory 24g --memory-swap 32g the system doesn't hang - instead the function fails & reconftw.sh moves on to the next thing. But, that's dependant on the output of this line so...

it would be probably better to report into LinkFinder repository

I was wondering, since reconFTW uses a fork of LinkFinder, should I open the issue there or on the original?

[six2dez]

Oh, interesting, thanks for debugging the issue.

Actually reconftw uses the default LinkFinder repository, not a fork, you can check on the installer script that it clones the GerbenJavado's LinkFinder repository.

[frost19k]

you can check on the installer script that it clones the GerbenJavado's LinkFinder repository

I'm looking at the script right now... line 86 repos["LinkFinder"]="dark-warlord14/LinkFinder" (same in dev)

[six2dez]

Ooooops :/ my bad, let me check why are we using this fork instead the original one

[frost19k]

Issue seems to be specific to this one domain. The script runs just fine on other targets.

Still have no idea what's causing it though, but I'm gonna go ahead & close this.