search

GerbenJavado / LinkFinder

A python script that finds endpoints in JavaScript files
https://gerbenjavado.com/discovering-hidden-content-using-linkfinder
MIT License
3.7k stars 595 forks source link

Linkfinder too fast to scan everythin? #79

Open Rdago opened 4 years ago

Rdago commented 4 years ago

Hello there,

I just ran Linkfinder on 393.162 JS-URLs and it took less than a second to run and it found 2 Endpoints. I dont think that it checked all of them. What am I missing?

My code: 

python3 linkfinder.py -i valid-js.txt -o cli
EdOverflow commented 4 years ago

What version of LinkFinder are you using?

Rdago commented 4 years ago

Hello @EdOverflow Latest version. I dumped everythin I had and pulled it again. Running it with python3 (if this matters)

Rdago commented 4 years ago

I guess it does not work with lists so I ran it like this: 

for i in `cat mydoms.txt`; do python3 linkfinder.py -i $i -o cli >>LF-out.txt; done

Moreover it would be nice if there where flags to just pull out absolute urls. I greped them manually like this: 

cat LF-out.txt | grep -E "http[s]?://.*" > LF-abs.txt
EdOverflow commented 4 years ago

Would you be willing to share a few lines of your input text file?

Rdago commented 4 years ago

Sure. 

mydoms.txt: 
https://github.githubassets.com/assets/environment-bootstrap-e3b72f9d.js
https://github.githubassets.com/assets/vendor-ed3c6d95.js
https://github.githubassets.com/assets/github-bootstrap-091a9cad.js

PS: glaube das tool kann stand jetzt keine Listen einlesen dementsprechend über ne For-Schleife gelöst, ist aber für meine Zwecke erstmal okey :)

danke für deine Hilfe @EdOverflow Ich sag mal bis zum CCC ;)

EdOverflow commented 4 years ago

You are absolutely right, @Rdago. There does appear to be a problem with lists now. Lists used to work so I must have broken it with an update. I will try my best to fix this issue as soon as possible.

Übrigens du kannst auch GNU Parallel verwenden falls du die Links in parallel einlesen möchtest. Lauft schneller als eine Für-Schleife. :)

$ parallel -j 12 "python3 linkfinder.py -i {} -o cli 2> /dev/null" :::: mydoms.txt