search

DanMcInerney / xsscrapy

XSS spider - 66/66 wavsep XSS detected
1.66k stars 441 forks source link

Added rate limiting. #15

Closed ddworken closed 10 years ago

ddworken commented 10 years ago

Added rate limiting in the form of a -r or --rate flag to set the number of requests per minute (using http://doc.scrapy.org/en/latest/topics/settings.html#download-delay). Changed something.com to example.com to comply with RFC 2606 (http://tools.ietf.org/html/rfc2606#page-2).

ddworken commented 10 years ago

Out of curiosity, why was this option removed?

IMO, there is a need for this in addition to the concurrent requests parameter. When scanning small websites, constantly maintaining even a single connection to the server would be noticeable compared to the normal load on the website. This can be avoided by limiting it to one connection per 30 seconds, which would not be noticeable compared to the normal load on the website.

DanMcInerney commented 10 years ago

I had figured concurrent_requests would serve the same purpose but I think you're right in that it serves a purpose so I readded it.