search

Qianlitp / crawlergo

A powerful browser crawler for web vulnerability scanners
GNU General Public License v3.0
2.83k stars 477 forks source link

Output Request Response to ".txt" File #97

Open gromhacks opened 2 years ago

gromhacks commented 2 years ago

First of all this tool is fantastic!

The primary reason I would like to switch to this tool is the DOM Rendering. Would you be able to incorporate a feature that outputs http.response to ".txt" files after the dom has been rendered? A lot of tools like httpx lack this feature and it would be super nice for indexing.

Qianlitp commented 2 years ago

The goal of crawlergo is to discover as many attack surfaces as possible, and the rendering process gets so much response data, while begging the question of which content types need to be saved, that it's not a simple feature.

What is the use of this feature for you? How does it help you with subsequent Web vulnerability scans?

undergroundwires commented 2 years ago

First of all thank you @Qianlitp , this is a great tool and I think the best crawling tool ever written according to my research, it will be much popular later.

I would also appreciate this functionality. Maybe not necessarily .txt files but in the output. It would be nice to get responseBody property like OWASP ZAP spider. This way, I can do further analysis on the web page content without needing to send one more request to get the response.