search

iSarabjitDhiman / TweeterPy

TweeterPy is a python library to extract data from Twitter. TweeterPy API lets you scrape data from a user's profile like username, userid, bio, followers/followings list, profile media, tweets, etc.
MIT License
145 stars 20 forks source link

get_tweet function with_tweet_replies can not return all the replies ? #50

Closed Unayung closed 6 months ago

Unayung commented 7 months ago

Hi, thanks for this amazing package 👍

Just one little issue here, I'm looking for a reliable approach to check all the replies from one tweet id. the replies count from twitter website like 3000-ish. but I'm only get 225 results returned.

I've tried pagination=True / False along with apply the end_cursor from last result returned but still. I'm also checked about rate limit which was false.

is it the limit of twitter graphql api limit or something ?

iSarabjitDhiman commented 7 months ago

Hey @Unayung

Thanks for reporting this issue. I just tested it with a single tweet and as you said the numbers don't match up. I was supposed to get 1.5K replies but I only got 210. I have my doubts, I just want to test a few things out before I come up with a solution.

Could you please share any of the tweet links/IDs that you have tested on your side? I want to verify something.

Please let me know.

Thanks.

Unayung commented 7 months ago

Thank you for the reply!

I got this tweet 1764315112729432555 around 1.4k replies, and 1764040593204818102 around 7k. and they all returned like 210 replies and that's it.

截圖 2024-03-04 11 22 45

iSarabjitDhiman commented 6 months ago

Hey @Unayung Sorry for the delay, I was kinda busy lately. Well, I just tested it out and I noticed something strange.

  1. Script is able to fetch all the replies available on the twitter.

    • Of course I didn't get all the 7K replies for this (1764040593204818102) tweet but I got around ~220 replies. Here is what I noticed, If you visit the same tweet on your web browser or on the twitter app and try to scroll down manually to get all the tweet replies, you will still get the same number of results as you received with the script/tool (~220 replies). So I can tell, there isn't any issue with the script.
      1. Twitter says that the post has 7k+ replies, I am trying to figure out what is going on in there. Maybe the number (7K) is the sum of the replies on the individual replies as well? Or maybe it also adds up (+) the number of replies received on reposts too?

    Please let me know if you find anything useful in the meantime. I will update my response as I conduct my research.

    Thanks.

Unayung commented 6 months ago

yep, I've noticed that even on the twitter website, we can not get all replies and match the number, weird. I guess it's just another un-documented change in the twitter API or something.

anyways, thank you :D

iSarabjitDhiman commented 6 months ago

yep, I've noticed that even on the twitter website, we can not get all replies and match the number, weird. I guess it's just another un-documented change in the twitter API or something.

anyways, thank you :D

Maybe the Twitter is restricting the number of results, just like Google does. If we search something on google, it will not let us get past 30th page or something, because no one except bots going to go past that many pages. I am not sure if twitter does the same. Just in case you find any solution to get past this limit, I would really appreciate if you could update/edit your answer. Also, feel free to close this thread if you have no further queries.