get_hashtag_tweets ERROR after loading for a while (Rstudio)

[Xibaaah]

i'm doing research on Twittersentiment from Tweets about 10 different stocks. So i need to collect Tweets and i'm using the "acadamictwitteR" package to collect them with the get_hashtag_tweets. So if i run this code it runs for a while as i need data tweets from 10 different stocks from 1/11/2019 untill 1/03/2021. After running for a while 500 per line (in the console) it just stops and gives me the 503 error, but everytime i check the servers of Twitter are online. Anybody that knows how i can fix this?

ps: i'm using Rstudio and i have an academic researcher account, i tried using the Rtweets package but i can't use the Search_fullarchive as you need a premium or enterprise account. Attached is the error i get after running for a while.

[eweisbrod]

I am having the same problem. So far the workaround I have found is to collect smaller and smaller data windows until the 503 error goes away. I usually collect from 2013-2020, but if a stock gives a 503, I start going year by year for that stock. If it still gives a 503, I go month by month, and so on. It seems to have something to do with a rate limit, even though the message says 503.

[Xibaaah]

Yes i'm trying the same but i need to go with for example 5 days as they already contain like 40K Tweets. So i'm getting there but after collecting the data i don't know how to merge the different file together from 1 stock as with the get_tweets() the columns are not always in the same order. So do you know how i could merge these files together of 1 stock and calculate only the sentiment on the Text box?

[cjbarrie]

One workaround would be to increase the sleep time when 503 errors are encountered. We're looking into this.

[eweisbrod]

@Xibaaah my situation is probably different because I wrote a function that takes the output from get_hashtag_tweets and splits it into many tables and stores each table in an sql database with the tweet ID as a key. With that said, if all that you need is some primary variables like the text of the tweet, then I think you could just get rid of all of the nested lists inside the output. If you do that, I think the part that is left would be the same variables for every search, and you could easily merge the dataframes together. Here is the part of my code where I select only the non-nested data if it helps:

output_list[["primary_tweet_data"]] <- dta %>% select(where(negate(is.list))) %>% as_tibble()

dta is the output of get_hashtag_tweets

[Xibaaah]

@eweisbrod i'm afraid your function does not work for me. It is probably a very simple function that i'm not familiar with but i need to get the Tweets i collected in one file per stock. After this is think i need to work with a vector to calculate the sentiment of this. (I have all the data now in several pieces but it have it, i also have a pos/neg lexicon for the sentiment already)

[cjbarrie]

503 errrors are not really an Issue on our side but on the client-side. Closing this Issue now but thanks for reporting.