Browsing through proxies

clemlesne / scrape-it-now

Web scraper made for AI and simplicity in mind. It runs as a CLI that can be parallelized and outputs high-quality markdown content.

Apache License 2.0

485 stars 15 forks source link

Closed clemlesne closed 1 week ago

clemlesne commented 2 months ago

Ideas:

Specify HTTP proxy + user/pass
Connect to Tor
Get proxy list (https://spys.me/proxy.txt, https://github.com/TheSpeedX/PROXY-List, https://github.com/monosans/proxy-list, https://github.com/proxifly/free-proxy-list) and connect
- Random usage?
- Disable temporary is not answering?
- If a request is denied from one, retry from another?
- Cache the list
Do we need to keep track of which proxy have been used for that scrape?

clemlesne commented 1 week ago

Aborted. Documentated in 04ab77c0e4d2ef5010506ee20280d5b95ff4b5db.