IP change

[artgod1]

Hi there , I am using Crawlera proxy rotation and I need to edit the settings.py. However, I don't see any option in that scraper to plug in proxy credentials. Can you pls help me with this ? Many Thanks

[tducret]

Hi @artgod1 ! Proxy management is not implemented yet. The client.py would need to be updated :

def _get(self, url):
        """ GET request with the proper headers """
        ret = self.session.get(url, headers=self.headers)  # <= Add proxies=proxyDict
        if ret.status_code != 200:
            raise ConnectionError(
                'Status code {status} for url {url}\n{content}'.format(
                    status=ret.status_code, url=url, content=ret.text))
return ret

Have a look at this > https://stackoverflow.com/a/8287752

I wondered if it would work without code modification by setting the proper environment variables. For example :

export HTTP_PROXY=10.10.1.10:3128
export HTTPS_PROXY=10.10.1.11:1080
export FTP_PROXY=10.10.1.10:3128

Could you please test it and tell us about it?

[artgod1]

sure mate, I will come back here later Thanks

[tducret]

Not much news here. I am closing the issue.