search

fhamborg / news-please

news-please - an integrated web crawler and information extractor for news that just works
Apache License 2.0
1.99k stars 414 forks source link

Proxy Server configuration (HttpProxyMiddleware) #234

Open bkrishnap opened 1 year ago

bkrishnap commented 1 year ago

Is there a way we can configure the proxy server with news-please? I would like to add a proxy endpoint to hide the main IP of the server that is installed with news-please.

I tried to figure out where I can fit it with the HttpProxyMiddleware with the help of below documentation and I am not successful with the configs.

https://doc.scrapy.org/en/1.1/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.httpproxy

fhamborg commented 1 year ago

did you check whether the requests library supports this?

bkrishnap commented 1 year ago

requests does support. Something like this? But where should be entrypoint for this ..

`import requests

proxies = { "http": "http://10.10.1.10:3128", "https": "http://10.10.1.10:1080", }

requests.get("http://example.org", proxies=proxies)`

mcnamara-charles commented 10 months ago

I'm curious about this as well.

SaadTazroute commented 1 month ago

Having the same question ? is there anyone who faced this problem ?