linvi / tweetinvi

Tweetinvi, an intuitive Twitter C# library for the REST and Stream API. It supports .NET, .NETCore, UAP (Xamarin)...
MIT License
1.01k stars 220 forks source link

Inconsistent userTimelineIterator results for same user #1113

Closed Brettyoke49 closed 3 years ago

Brettyoke49 commented 3 years ago

I'm trying to retrieve up to 1500 original tweets for a given twitter username using a snippet of code that looks a lot like this:

public async int GetUserTweets(string userHandle)
{
            IUser user;
            try
            {
                user = await appClient.Users.GetUserAsync(userHandle);
            }
            catch (Exception ex)
            {
                //Write exception to logger, usually user doesn't exist or is private
            }

            ITwitterIterator<ITweet, long?> userTimelineIterator = appClient.Timelines.GetUserTimelineIterator(user.Id);
            int count = 0;

            while (!userTimelineIterator.Completed && count <= 1500)
            {
                var page = await userTimelineIterator.NextPageAsync();

                foreach (var tweet in page)
                {
                    if (String.Equals(tweet.CreatedBy.ScreenName.ToLower(), userHandle) && !tweet.IsRetweet)
                    {
                        count++;
                    }
                }
            }
            return count;
}

If I pass in user 'elonmusk' to this method, I get answers ranging from 0, to 192, to 760, to 1500. It's random each time. When I debug, it's simply a matter of sometimes the userTimelineIterator variable is marked 'completed' earlier than others.

linvi commented 3 years ago

That is correct and this is why the iterators were created. Twitter API paging system does not provide consistency over the number of results returned.

Tweetinvi takes care of all the experienced edge cases that you can encounter with pages in Twitter and provide a common pattern to iterate over the results. For some endpoints Tweetinvi iterators takes care of consistency by making additional requests to ensure that 0 results are returned.

Twitter API paging is very complex and does not provide consistency over the results on some endpoints. I strongly suggest you use the iterators to manage pages.

You can learn more about iterators here: https://linvi.github.io/tweetinvi/dist/twitter-api-v1.1/iterators.html?highlight=page#iterators