mahrtayyab / tweety

Twitter Scraper
480 stars 64 forks source link

Live tweet monitoring #159

Closed Jamie1605 closed 9 months ago

Jamie1605 commented 9 months ago

What would be the most efficient way to monitor new incoming tweets live?

Currently I run a search, save each tweet to a list, and then repeat the same search every few seconds, adding any new tweet that isn't already in the list, but that leads to a rate limit error pretty quickly if I want to do it reliably.

Is there a more efficient way to achive this that I'm missing?

mahrtayyab commented 9 months ago

Right now try to multiple accounts together if you want to do it every second, furthur search has 50 requests per 15 minutes rate limit , try using Twiiter List

Jamie1605 commented 9 months ago

Is a 'request' one tweet or one search i. e. would it help to limit the searches to a smaller amount of tweets?

mahrtayyab commented 9 months ago

one search request

Jamie1605 commented 9 months ago

So the amount of tweets is irrelevant to the rate limit?

mahrtayyab commented 9 months ago

making one request retrieve +-20 tweets.

Jamie1605 commented 9 months ago

And how exactly ist the tweety list different from a normal python list?

mahrtayyab commented 9 months ago

You mean python list

Jamie1605 commented 9 months ago

Right now try to multiple accounts together if you want to do it every second, furthur search has 50 requests per 15 minutes rate limit , try using Twiiter List

What did you mean by "Twitter List" here? Maybe I misunderstood

mahrtayyab commented 9 months ago

https://twitter.com/<username>/lists, You can create a a custom timeline for specific users , it is called List on Twitter

Jamie1605 commented 9 months ago

The purpose of my code is to archive tweets of one specific user if they are deleted using app.search('from:username', filter_=SearchFilters.Latest()) so a list probably won't make sense for me

mahrtayyab commented 9 months ago
  1. Create a list
  2. add specific user in that list
  3. keeping checking for tweets in the list using get_list_tweets
mravlincic commented 9 months ago
  1. Create a list
  2. add specific user in that list
  3. keeping checking for tweets in the list using get_list_tweets

Function get_lists_tweets() has 50 requests per 15 minutes rate limit? Is there also 30 day limit for it?

Jamie1605 commented 9 months ago
  1. Create a list
  2. add specific user in that list
  3. keeping checking for tweets in the list using get_list_tweets

Ah, okay. I still don't quite get how the way to do it with the list will be better for the rate limit than the normal search.

mahrtayyab commented 9 months ago
  1. Create a list
  2. add specific user in that list
  3. keeping checking for tweets in the list using get_list_tweets

Function get_lists_tweets() has 50 requests per 15 minutes rate limit? Is there also 30 day limit for it?

get_list_tweets has upto 500 request per 15 minutes rate limit

mahrtayyab commented 9 months ago
  1. Create a list
  2. add specific user in that list
  3. keeping checking for tweets in the list using get_list_tweets

Ah, okay. I still don't quite get how the way to do it with the list will be better for the rate limit than the normal search.

get_list_tweets has upto 500 request per 15 minutes rate limit and also returns more than 60 tweets in one request as compared 20 in search

Jamie1605 commented 9 months ago
  1. Create a list
  2. add specific user in that list
  3. keeping checking for tweets in the list using get_list_tweets

Ah, okay. I still don't quite get how the way to do it with the list will be better for the rate limit than the normal search.

get_list_tweets has upto 500 request per 15 minutes rate limit and also returns more than 60 tweets in one request as compared 20 in search

Do you know how many requests tweet_detail has? Thats the function I use to check if a tweet has been deleted. I run it for every tweet and if a tweety.exceptions_.InvalidTweetIdentifier occurs I know it was deleted. I assume that plays a role too.

And if it's too heavy, is there a better way to check if a tweet has been deleted?