twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.86k stars 2.73k forks source link

use twint to get some data, but Error retrieving https://twitter.com/ #1074

Open WjcSoso0928 opened 3 years ago

WjcSoso0928 commented 3 years ago

Dear all when I use twint demo to get twitter data, it get me a error

this is my code:

import twint

c = twint.Config()
c.Username = "realDonaldTrump"
c.Search = "great"
c.Proxy_host = "127.0.0.1"
c.Proxy_port = 1080
c.Proxy_type = "Socks5"

twint.run.Search(c)

I use proxy to connet,because I'm in China you know have some prohibit reason

this is error code:

WARNING:root:Error retrieving https://twitter.com/: ConnectionError(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fc6145800b8>: Failed to establish a new connection: [Errno 101] Network is unreachable',))",),), retrying

I can get the Google and Youtebe pages and also I can get tweepy data with this proxy but can't use this porxy to get twint data I have no ideal Thanks for help

innocentius commented 3 years ago

Have you tried to delete the proxy settings in your code? If you are already connected with some sort of TAP device then you don't need any proxy settings here.

WjcSoso0928 commented 3 years ago

Have you tried to delete the proxy settings in your code? If you are already connected with some sort of TAP device then you don't need any proxy settings here.

em... I tried this method, but it's still raise the same error

innocentius commented 3 years ago

What Proxy software are you using?

DoctorDream commented 3 years ago

What Proxy software are you using?

I am in China too and I use the ClashR for Win as my proxy software.I have the same problem like WjcSoso0928 and whether I add proxy settings or not, this error is raised.

WARNING:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x000001DF7FF3D248>, 'Connection to twitter.com timed out. (connect timeout=10)'))")), retrying

Yesterday, I use the command "pip install twint" to install twint and use the code like below.

c = twint.Config()
c.Username= "GenshinImpact"
c.Store_object = True
c.Limit = 100
c.Proxy_host = "127.0.0.1"
c.Proxy_port = 7891
c.Proxy_type = "Socks5"
twint.run.Search(c)

And I can get the error CRITICAL:root:twint.run:Twint:Feed:noDataExpecting, and I am sure that this error meanings the proxy setting works, because i can't get any reply before i use this proxy. I find out that use the newest github repository to install twint will resolve this CRITICAL and I do so. Then the error become the problem I mentioned above.

I noticed that maybe the parameter Tor_control_port become 9051 differs from the previous versions. Do you have a solution?Thank you very much!

innocentius commented 3 years ago

@DoctorDream @WjcSoso0928 I'm using ClashR for Win as well and I don't get any of the errors you guys are talking about. Instead of setting proxy here I turned ClashR TAP mode on, and no errors like that pops up.

Try to use TAP mode first, if not we could come up with something else.

zeminglakuai commented 3 years ago

你们解决了吗?我也是这样的问题