search

davidbauer / Instacurate

Turn your Twitter timeline into a personalised news site, in an instant. Fetches links from your timeline and displays them in a discovery friendly design.
http://www.instacurate.com
113 stars 29 forks source link

Call search api multiple times if not enough links were found (fixes #51) #58

Closed pfigel closed 11 years ago

pfigel commented 11 years ago

After parsing the Search API results the number of links will be checked and if there aren't enough links yet, we'll try to get more tweets. We're using Search API's max_id param to retrieve only tweets tweeted before the last results (i.e. paginate over the results).

Search API requests are limited to 10 for every hashtag right now as we don't want to send too many requests (this is configurable through maxSearchApiRequests)

davidbauer commented 11 years ago

Thanks a lot.

pfigel commented 11 years ago

It seems that for some hashtags, fetching 10 * 100 tweets won't give us enough links to work with.

I've found another solution - it's possible to add a filter to the Search API call so it only returns tweets with links. Pull request is on its way. At this point the code responsible for paginating through the search results is not really needed anymore, tho it's still (theoretically) possible that the first 100 tweets all contained the same link (or at least less than the number of links we need to render the page), so I guess it doesn't hurt anybody to leave it in there.

davidbauer commented 11 years ago

Thanks. I think the real issue is that the Search API only gives you results from a few days back (I think I once read something about t-7 days), so that of course limits the number of links you can get, depending on the hashtag.