search

janreges / siteone-crawler

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
https://crawler.siteone.io/
MIT License
255 stars 17 forks source link

Proxy support #1

Closed fernstedt closed 1 year ago

fernstedt commented 1 year ago

Please allow --proxy=IP option to add proxy or allow for a file with proxys or even pipe thu TOR

janreges commented 1 year ago

Hi,

now you can use --proxy=host:port.

Thanks for the idea to use more proxies. I can add --proxy-list=IP1,IP2,... and --proxy-list-file=proxies.txt.

Do you also have a concrete idea how this should work in case of defining more proxies?

I can think of for example the possibility of --proxy-strategy with the value random or round-robin and some directive that would control the behavior in case it would be necessary to retry the request with another proxy due to a broken proxy.