search

rverton / webanalyze

Port of Wappalyzer (uncovers technologies used on websites) to automate mass scanning.
MIT License
955 stars 137 forks source link

Crawl functionality #15

Closed anotherdevs closed 5 years ago

anotherdevs commented 5 years ago

Hi @rverton

Is it possible to add a crawl functionality that crawls fx. the first 20 (configurable) found sub pages?

Wappalyzer has these variables available: maxDepth maxUrls maxWait recursive

rverton commented 5 years ago

I think this can be added easy. Will think over this and let you know.

rverton commented 5 years ago

This was a bit more work than I thought. I added a waitgroup for the job channel so I can feed the new URLs to the worker queue. Additionally, comparing URLs can be a bit tricky. There is no depth setting currently and it is only crawling from the initial page load for now. Let me know if this works for you.