search

jonluca / Anubis

Subdomain enumeration and information gathering tool
https://jonlu.ca/anubis/
MIT License
1.18k stars 151 forks source link

Consider using selenium instead of requests + regex #16

Closed KOLANICH closed 6 years ago

jonluca commented 6 years ago

Using a headless browser would add quite a bit of overhead (and make it considerably slower), and would complicate the install process. Provided the pages Anubis is scraping don't use js to fetch data or use a framework, the results should be identical.

Why do you think I should use selenium (genuinely asking)?

KOLANICH commented 6 years ago

Because if you mimic browser you are trying to be stealthy. If you wanna be stealthy, there is no better mimic of a browser than actula browser.

And selenium is not headless. It's just a library implementing the api side of the webdriver standard. On the other side is usual desktop web browser.

jonluca commented 6 years ago

I don't think the overhead is worth it. It's possible to revisit this in the future, but for now requests + regex is good enough.