twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.66k stars 2.72k forks source link

Error when trying to multi-thread the twint.Followers() method #109

Closed jo1989 closed 6 years ago

jo1989 commented 6 years ago

Initial Check

Make sure you've checked the following.

If this is a feature request please specify in the title:

Example:

[REQUEST] More features!

Command

Please provide the exact command ran including the username/search so I may reproduce the issue.

Description of Issue

Hey, first of all Nice work. Then, I would know if it is possible to multi-thread or multi-proc the twint.Followers() / Friends() methods. Because, I try to do it with python code but got this error :

Exception in thread Thread-167:
Traceback (most recent call last):
  File "C:\Users\Jo\Anaconda3\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\Jo\Anaconda3\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-65-b2613fd19b4c>", line 6, in work
    twint.Followers(item)
  File "C:\Users\Jo\Anaconda3\lib\site-packages\twint\followers.py", line 22, in __init__
    loop = asyncio.get_event_loop()
  File "C:\Users\Jo\Anaconda3\lib\asyncio\events.py", line 694, in get_event_loop
    return get_event_loop_policy().get_event_loop()
  File "C:\Users\Jo\Anaconda3\lib\asyncio\events.py", line 602, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Thread-167'.

Thanks.

OS Details

Using Windows, Linux? What OS version? Windows 10

pielco11 commented 6 years ago

Well, this seems not to be related to Twint. And if you don't post code I don't know how one can help you

Btw, if you would like to run multiple threads to scrape followers of a single user, I don't think that this could work: the function is waiting for data, adding multiple threads will not speed up since the "slowness" is server side

Instead you could run different threads for different users defining different Twint instances, one for user, and run them in separated threads. A more complicated solution is implementing multi-threading in Twint (which is quite interesting and could be a nice feature)

Hope this helps you somehow (closing this because is not related to Twint)

Nestor75 commented 6 years ago

I had the same error and I fixed it following this post https://github.com/tornadoweb/tornado/issues/2308 . This is the one I opened and closed https://github.com/twintproject/twint/issues/171

pielco11 commented 6 years ago

171