Open dtrillo opened 3 years ago
Hello! What is the error that requires the use of a proxy?
Unfortunately using proxies isn't a feature that is supported out of the box but you can probably brew up your own solution using requests
builtin proxy support. instascrape
can scrape data from HTML so you can use requests
(with your needed proxies) to request data and then pass the response's HTML into an instascrape
scraper to scrape the data.
Yes, the point is to prepare "something" that you can add to request
. I did it in other projects but the problem within instascrape
is that I don't know how to acccess request
to set my proxy credentials.
@chris-greening Got some code working to scrape and validate proxies, but am not really sure how your library works. Willing to collaborate on this?
@dtrillo We had to solve this problem as well. What I ended up doing was subclassing the Session
object to pass in to Profile.scrape
:
class Session(requests.Session):
"""Custom session subclass that allows passing through a timeout to requests.get"""
def __init__(self, timeout=None, proxies=None):
super().__init__()
self.timeout = timeout
self.proxies = proxies
def get(self, url, **kwargs):
kwargs.setdefault('allow_redirects', True)
return self.request('GET', url, **kwargs, timeout=self.timeout, proxies=self.proxies)
and then using it this way:
with Session(timeout=20, proxies=self.proxies) as s:
profile.scrape(session=s)
The above is only necessary for us for authenticated proxies if you want to keep trust_env = True
for the Session. Otherwise you can just create a session and add proxies to that without the need for the subclassing
@chris-greening . I am trying to use proxies with InstaScrape to avoid blocking. I tried the solution of @nickhendo but can't get it to work. Any ideas?
I am trying InstaScrape but I need to set a proxy (otherwise I have an error). How do I have to proceed?