search

iSarabjitDhiman / TweeterPy

TweeterPy is a python library to extract data from Twitter. TweeterPy API lets you scrape data from a user's profile like username, userid, bio, followers/followings list, profile media, tweets, etc.
MIT License
138 stars 20 forks source link

Enhancements: List tweets and data models #1

Closed codilau closed 1 year ago

codilau commented 1 year ago

Hey there, @iSarabjitDhiman, Congratulations for picking this subject in these turbulent times and also for the dynamic handling of the qraphql endpoints which is something I didn't see in other implementations. I'd like to suggest implementing the ListTweets endpoint if you can find the right documentation for that (i've found some conflicting info). Also, something useful for people looking for authenticated scrape is using data models such as those implemented by SNScrape.twitter module. This would make switching to TweeterPy easier for a lot of people. Thanks!

iSarabjitDhiman commented 1 year ago

Hi @codilau , thanks for showing your interest in this project. By ListTweets, do you mean user tweets from the user's profile? If yes, its has already been implemented, check get_user_tweets method in the docs. If its something else, please feel free to share some reference or examples. I will try to add it to the very next update.

About data models, sounds good. I will surely take a look at the sns scrape library.

Thanks.

codilau commented 1 year ago

Here's the link to the SNScrape implementation, with all credit to JustAnotherArchivist for the brilliant work, SNScrape will not support authentication as he stated himself. https://github.com/JustAnotherArchivist/snscrape/blob/master/snscrape/modules/twitter.py In regards to the List, that's a very useful feature since it allows following curated content with up to 5k profiles, so one call to that endpoint could worth up to 5k calls to user_tweets. Here's an exemple of a list URI: https://twitter.com/i/lists/1338136630062886912 I can't be sure, the info should be somewhere close but that endpoint might be this one: /graphql/ZBbXrl37E6za5ml-DIpmgg/ListLatestTweetsTimeline

iSarabjitDhiman commented 1 year ago

Here's the link to the SNScrape implementation, with all credit to JustAnotherArchivist for the brilliant work, SNScrape will not support authentication as he stated himself. https://github.com/JustAnotherArchivist/snscrape/blob/master/snscrape/modules/twitter.py In regards to the List, that's a very useful feature since it allows following curated content with up to 5k profiles, so one call to that endpoint could worth up to 5k calls to user_tweets. Here's an exemple of a list URI: https://twitter.com/i/lists/1338136630062886912 I can't be sure, the info should be somewhere close but that endpoint might be this one: /graphql/ZBbXrl37E6za5ml-DIpmgg/ListLatestTweetsTimeline

Hey, I guess you are talking about Tweet Lists from the Twitter Mobile App? The endpoint and the url you shared above, took me to twitter's lists page. I just added get_list_tweets method, check and test if this is what you meant. I will add more endpoints and check this SNSScrape library as soon as I finish working on (async/await) concurrent requests feature.