search

TeamHG-Memex / scrapy-rotating-proxies

use multiple proxies with Scrapy
MIT License
732 stars 156 forks source link

Adding tor socks proxy for onion sites #6

Open abuadan opened 7 years ago

abuadan commented 7 years ago

I was wondering if there is a way to specifiy to scrapy_rotate_proxy to use socks5h proxy for .onion sites

kmike commented 7 years ago

@abuadan Scrapy doesn't have socks5 proxy support (https://github.com/scrapy/scrapy/issues/747), so the way to go is to setup a regular proxy in front of socks5; then you can use it as usual with scrapy-rotating-proxies. If you want to use such proxy for .onion links, and rotate proxies for other links, you can set request.meta['proxy'] explicitly for .onion links.