direct_messages_received (and sent) immediately prompts Twitter::Error::TooManyRequests

[joshkestenberg]

Code in question: messages = @client.direct_messages_received(count: 25)

Initially I thought that maybe I was facing issues when iterating over the collection as per issue #747 but it appears that this error happens right when I make this request. Is there something at else at play here? It seems like from this call alone, based on tracing the stack, there's no way I could be making more than one request. Any insight would be appreciated.

At first I thought that maybe the user in question had some other Twitter integration app that was eating up their rate limit, and I'm starting to lean in that direction again. However, this is only happening to two users in my database, both of whom appear to be very active twitter users - other users are able to make this call without hitting the rate limit - so I'm wondering if this is indeed prompted by attempting to somehow fetch a large quantity of DMs.

[joshkestenberg]

The issue appears to be that the API keeps making cursor requests until it's fetched the amount of messages specified by count, but the number of results on each page varies dramatically, so for some users, count: 25 could mean 1 request, and for another, it could mean 16 requests (where page 1 has 3 results, page 2 has 0, page 3 has 5, etc..) ie. until the rate limit is hit.

I don't understand why the pages are all different lengths though. From twitter:

The Twitter standard REST APIs utilize a technique called ‘cursoring’ to paginate large result sets. Cursoring separates results into pages (the size of which are defined by the count request parameter) and provides a means to move backwards and forwards through these pages.

In reality, however, it doesn't seem like the count parameter has any effect on how many results we get per page..

I'm stumped.

[AgoraSecurity]

I know Twitter has a known bug regarding the results of each page: https://twittercommunity.com/t/dm-bug-next-cursor-returned-with-an-empty-events/130286/2

Anyhow it's quite annoying the lack of simple control of how many requests will be made. I understand you can control with the option "limit" https://github.com/sferik/twitter/blob/master/lib/twitter/cursor.rb#L29

Note: This is also a problem for me. I'll keep researching and post an update when I have one.

[GCorbel]

@sferik I have the issue with the latest version. I think we can stop to continue the loop when there are no more events.

Twitter::Enumerable#each can be changed to :

  def each(start = 0, &block)
    return to_enum(:each, start) unless block_given?

    Array(@collection[start..]).each(&block)
    unless finished?
      start = [@collection.size, start].max
      next_page = fetch_next_page
      each(start, &block) if next_page[:events].any?
    end
    self
  end

I assume once the API returns an empty array, next pages will always give empty arrays too.