[Feature]: Support crawling through pre-configured SOCKS5 proxies

[ikreymer]

Context

There are several scenarios where it may be beneficial to crawl through a more distributed network of nodes, besides the ones where the crawl is running. Distributing K8s infrastructure is tricky, however we can easily have the crawler crawl through designated SOCKS5 proxies.

What change would you like to see?

While there are many possible use cases, the initial goal is to support a user that wants to:

1. crawl content as seen from a particular geographic region, due to geolocation restrictions or other requirements.

Additionally, we may also want to support a user that wants to:

2. Get around rate-limiting, which may happen form extensive crawling from a single IP.
3. Crawl through a designated IP on their own infrastructure with provided credentials / ssh key
4. Crawl through a randomly assigned IP via the Tor network and/or crawl Tor content.

Requirements

The core requirements if for:

• crawler to support configuring a SOCKS5 proxy (implemented in Browsertrix Crawler)
• cluster-wide configures of SOCKS5 proxies via the helm chart, similar to storages (use case 1).
• ability to select a proxy from available proxies as part of the crawl config.

Additional requirements will be added for use cases 2-4 as needed.

3) will require options to configure custom proxy settings. 4), if implemented, may require using Tor support in Brave and/or running separate Tor proxy. 2) may involve additional rate limiting detection, or other failover mechanisms, and will require some R&D.

Todo

The initial focus of this will be to support use case 1), crawling through a list of preconfigured proxies. Will involve:

• Support for SOCKS5 via Browsertrix Crawler
• Backend support for managing proxies
• Frontend support for selecting a proxy from available options.

[tuehlarsen]

when do you expect to implement this feature? At our site it's necessary for all paywall restricted sites such as newssites and similar ip restricted webplatforms - we try to archive with Heritrix today.

[tw4l]

Hi @tuehlarsen, it's something we hope to get to before too long, but we don't currently have it roadmapped for a particular release. We will update the issue when it is being worked on.

[tw4l]

@tuehlarsen You'll be happy to know that @vnznznz is working on our first implementation of crawling through SOCKS5 proxies now. At first stage it will likely just be country-specific, but later on we should be able to support local proxies to crawl from specific IPs.