search

hakluke / hakrawler

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
https://hakluke.com
GNU General Public License v3.0
4.49k stars 497 forks source link

Add round-robin proxy switcher #80

Closed gohumble closed 3 years ago

gohumble commented 4 years ago

Adds support for colly's round-robin proxy switcher.

Thought this might be useful.

Might do the job for #63 but not sure about #43 (not replaying requests).

Tested this using tor relay proxy.

gohumble commented 4 years ago

oh i see that my linter also fixed some indents in collector.go could undo this if you want.

hakluke commented 3 years ago

hey sorry I've rewritten the whole thing now!

gohumble commented 3 years ago

But still not support for colly.RoundRobinProxySwitcher :( Going to open a new PR to the rewritten version and add the RoundRobinProxySwitcher if you are ok with that.