miatang13 commented 4 years ago

I have the same issue. "twitterscraper Trump --limit 1000 --output=tweets.json" gives an empty json file.

calebgrisell commented 4 years ago

same issue :(

kaushalvivek commented 4 years ago

Same issue, getting 0 tweets.

Aivi001 commented 4 years ago

Yes, I'm having the same problem, getting the following error message when "getting 0 tweets"

raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: /search?f=tweets&vertical=default&q=%23Harami%20since%3A2020-08-22%20until%3A2020-08-24&l=en (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5a3decb510>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

GivenToFlyCoder commented 4 years ago

I have the same issue, 0 tweets using query_tweets...

jason1991220 commented 4 years ago

It doesn't work and return 0 tweet again. please help.

christiangfv commented 4 years ago

ooh bad, doesn't work function query_tweets from_user() nor query_tweets(), since yesterday. please help. maybe here is information https://blog.twitter.com/developer/en_us/topics/tips/2020/understanding-the-new-tweet-payload.html

gautampal1947 commented 4 years ago

0 Tweets again. Please help.

zhicheng0501 commented 4 years ago

ooh bad, doesn't work function query_tweets from_user() nor query_tweets(), since yesterday. please help. maybe here is information https://blog.twitter.com/developer/en_us/topics/tips/2020/understanding-the-new-tweet-payload.html

The same error. I agree with you.

tpsvld commented 4 years ago

Easy Twitter enabled Javascript and shut down the backdor this code used In detail: Twitter has been developing since beginning of 2020 a new HTML Frontend with Randomly generated HTML Description Tags. The scoll mechanism to load new tweets was already changed to work with javascript and unique session ID's in january 2020. The URL "https://twitter.com/i/search/ with the &max_position=cursor" has been disabled in twitter.

It was surprising that with the russia collusion and other horse shit they didnt start to work to protect the data.

So as i said in April, the software is probably dead.

tpsvld commented 4 years ago

ooh bad, doesn't work function query_tweets from_user() nor query_tweets(), since yesterday. please help. maybe here is information https://blog.twitter.com/developer/en_us/topics/tips/2020/understanding-the-new-tweet-payload.html

This doesnt work with the Twitter API not even close.

They changed the HTML Page to avoid non conscented web scrapping.

AveryData commented 4 years ago

So is this library just donzo? It's over?

zhicheng0501 commented 4 years ago

So is this library just donzo? It's over?

Temporarily. Yeah. Hope it will be back soon

pavstar619 commented 4 years ago

Thought I'd use this for my project. Guess that's not gonna happen anytime soon :(

edmangog commented 4 years ago

Same here. Trying the branch of selenium alternatively.

sudz4 commented 4 years ago

Is there another good similar library that people are using?

adamzvx commented 4 years ago

same to me

Simon-Smith-NKU commented 4 years ago

same issue ，twitter disgust,

qarampage commented 4 years ago

Even I got 0 Tweets count. I posted a new tweet and still got Zero tweets. I spent a lot of time understanding the workflow and got 0 !

sudz4 commented 4 years ago

just use tweepy instead

import tweepy as tw

Importing twitter access keys from a separate file in this project folder:

from credentials import * #Using keys as a variable

Setting up Twitter API connection:

def twitter_creds(): """ Connects to Twitter API. For confidentiality purposes the access keys are in a separate file in the project folder. Credential file is saved as credentials.py.

"""

#Authenticate credentials using keys:
auth = tw.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET) #tells twitter I am valid user
auth.set_access_token(ACCESS_TOKEN, ACCESS_SECRET)

#Connect with API and authenticate:
api = tw.API(auth) #speaks to twitter
return api

Finding some AI Tweets

search_words = "artificial+intelligence" date_since = "2020-1-1"

2020-3-12 outputted 669 tweets

For fun let's see if we can get a little closer to 1000

2020-3-12 gives me 673 tweets. 4 tweets in 3 months??? I don't know...

2020-1-1 gets us to the limit set below of 1000

tweets = tw.Cursor(api.search, q=search_words, lang="en", since=date_since).items(1000)

for tweet in tweets: print(tweet.text)

do-me commented 4 years ago

As twitterscraper is (temporarily) dead, I thought I'd share my favorite alternative scraper. @mattwsutherland your solution requires an API-key (which is super hard to get) so unfortunately it's useless to most people here.

Simply go for TweetScraper - for me its working perfectly. Also see this medium article providing a useful pandas snippet to set up a nicely formatted dataframe.

Installation for Ubuntu is straightforward as described. For Windows just do the following:

git clone https://github.com/jonbakerfish/TweetScraper.git
conda create -n tweetscraper python=3.7.7 -y
conda activate tweetscraper
conda install -y -c conda-forge scrapy ipython ipdb
pip install scrapy-selenium

Change to directory "TweetScraper" and test with

scrapy crawl TweetScraper -a query="foo,#bar"

qarampage commented 4 years ago

Thanks I will definitely 👍 try that

On Wed, Nov 11, 2020, 1:40 AM do-me notifications@github.com wrote:

As twitterscraper is (temporarily) dead, I thought I'd share my favorite alternative scraper. @mattwsutherland https://github.com/mattwsutherland your solution requires an API-key (which is super hard to get) so unfortunately it's useless to most people here.

Simply go for TweetScraper https://github.com/jonbakerfish/TweetScraper

for me its working perfectly. Also see this medium article https://medium.com/@kevin.a.crystal/scraping-twitter-with-tweetscraper-and-python-ea783b40443b providing a useful pandas snippet to set up a nicely formatted dataframe.

Installation for Ubuntu is straightforward as described. For Windows just do the following:

git clone https://github.com/jonbakerfish/TweetScraper.git conda create -n tweetscraper python=3.7.7 -y conda activate tweetscraper conda install -y -c conda-forge scrapy ipython ipdb pip install scrapy-selenium

Change to directory "TweetScraper" and test with

scrapy crawl TweetScraper -a query="foo,#bar"

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/taspinar/twitterscraper/issues/344#issuecomment-724939061, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD4DDQIMFDUXIU7TEKI5O73SPGM4HANCNFSM4REYB7DQ .