search

TeamHG-Memex / scrapy-rotating-proxies

use multiple proxies with Scrapy
MIT License
733 stars 156 forks source link

how to ban the proxy with timeout? #65

Open leolle opened 2 years ago

leolle commented 2 years ago

I have the global DOWNLOAD_TIMEOUT in setting for 3 seconds, how to ban the proxy if timeout happens? Thank you.

PS, I can see the proxy with connection timeout is active all the time.

cpouldev commented 2 years ago

@leolle add this method to your spider

from twisted.internet.error import TimeoutError

def exception_is_ban(self, request, exception):
        return isinstance(exception, TimeoutError)