Iterating over search results for Twitter v1.1 API (or any other previously paginated result)

[ptwobrussell]

With v1.1 of the Twitter API, there is no concept of pagination for queries such as search. The way that you get the next batch of results is to parse out the response and index into the search_metadata['next_results'] parameter which will be a value such as '?max_id=313519052523986943&q=NCAA&include_entities=1'.

Is the long term intention for the user of this package to parse that out and pass back in those parameters in a looping construct as application specific logic or did you have any plans or thoughts on making it more trivial to navigate through search results?

Parsing out the parameters for the next page and issuing multiple calls in a looping context is easy enough to do, but it could be interesting to consider ways to make getting the next page of results through this or any other API that previously had the concept of pagination with a construct such as a lazy iterator that could be sliced so that you could effectively issue a query like

twitter_search.search.tweets(q="NCAA")[:5] #get 5 pages

or

results = []
search_iterator = twitter_search.search.tweets(q="NCAA")
for i in range(1,6):
    results.append(search_iterator())

The exact API is worth thinking about some more, but I think that gets the general suggestion across.

[sixohsix]

Hi,

The current API maps almost 1-to-1 with the URLs one accesses at Twitter to get data. It has no opinion about the meaning of the data it receives. In your suggested feature, the API now has to understand returned data and change behaviour depending on this.

Of course, the behaviour you suggest is highly valuable. I would prefer to put it in a helper function:

search_iterator = iter_results(twitter_search.search.tweets, q="NCAA")

iter_results would repeatedly call the first argument with the given kwargs, augmented with the pagination args.

At this time I don't plan to implement this, but I would welcome a pull request with the feature. :)