search

energyapps / social-scraper

Scraper scripts that collect social media information for energyapps.github.io/social
0 stars 2 forks source link

Unknown twitter error when twitter.py running on local machine #2

Open DanielJWood opened 7 years ago

DanielJWood commented 7 years ago

When running the platforms_mini/twitter.py script locally, the following error is produced (see comment). However, when this script is run on the EC2 instance, no such error is produced.

NOTE This same code and errors featured in hourly_follower_count/followers_hourly.py and org_chart_data/master.py.

I think it might be related to throttling one IP address vs. another, but I have not had the ability to research it in depth. No such error is found on the very similar youtube or instagram script.

@chaunceyt When copying the two main python codes into the new Jenkins environment, care should be take to ensure the twitter scraper (which uses python packages requests and BeautifulSoup) is functioning properly.

DanielJWood commented 7 years ago 
Traceback (most recent call last):
  File "twitter.py", line 6, in <module>
    response = requests.get(url)
  File "/Library/Python/2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/adapters.py", line 487, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: /justinbieber (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x104e5c550>: Failed to establish a new connection: [Errno 61] Connection refused',))