求scrapy中间件 - Githubissues

SpiderClub / haipproxy

:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

https://spiderclub.github.io/haipproxy/

MIT License

5.43k stars 914 forks source link

求scrapy中间件 #44

Open gsoec opened 6 years ago

ResolveWang commented 6 years ago

这个需求您可以直接使用py_cli.py进行对接啊

yuxiaorui commented 5 years ago

@gsoec 用py_cli.py里面的ProxyFetcher，不到10行代码就可以实现。

from haipporoxy.client import ProxyFetcher

class CustomProxyMiddleware(HttpProxyMiddleware):
    fetcher = ProxyFetcher('zhihu', strategy='greedy', length=5)

    def process_request(self, request, spider):
        proxy = self.fetcher.get_proxy()
        print "**************ProxyMiddleware no pass************" + proxy
        request.meta['proxy'] = self.fetcher.get_proxy()