twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.81k stars 2.73k forks source link

[REQUEST] useragentstring.com is down, add a fallback #852

Closed gridsquare closed 4 years ago

gridsquare commented 4 years ago

Issue Template

Please use this template!

Initial Check

If the issue is a request please specify that it is a request in the title (Example: [REQUEST] more features). If this is a question regarding 'twint' please specify that it's a question in the title (Example: [QUESTION] What is x?). Please only submit issues related to 'twint'. Thanks.

Make sure you've checked the following:

Command Ran

os.environ['TWINT_DEBUG'] = 'debug'

import twint

c = twint.Config()
c.Username = "twitter"

twint.run.Favorites(c)

Description of Issue

when useragentstring.com is down, twint.run.Favorites(c) doesn't returning anything head of the twint.log:

DEBUG:2020-08-03 14:51:34,140:root:twint.run:Favorites
DEBUG:2020-08-03 14:51:34,140:root:twint.run:run
DEBUG:2020-08-03 14:51:34,140:asyncio:Using selector: KqueueSelector
DEBUG:2020-08-03 14:51:34,140:root:twint.run:Twint:__init__
DEBUG:2020-08-03 14:51:34,140:root:twint.datelock:Set
DEBUG:2020-08-03 14:51:34,142:root:twint.run:Twint:__init__:pandas_clean
DEBUG:2020-08-03 14:51:34,142:root:twint.get:RandomUserAgent
DEBUG:2020-08-03 14:51:39,490:fake_useragent:Error occurred during fetching http://useragentstring.com/pages/useragentstring.php?name=Chrome
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/socket.py", line 808, in create_connection
    raise err
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/socket.py", line 796, in create_connection
    sock.connect(sa)
socket.timeout: timed out

Environment Details

Using Windows, Linux? What OS version? Running this in Anaconda? Jupyter Notebook? Terminal? Mac zsh

gridsquare commented 4 years ago

Fallback already implemented