search

scraperwiki / twitter-search-tool

ScraperWiki tool to get Tweets matching a search term; tool now defunct, though the code is here for reference.
https://blog.scraperwiki.com/2014/08/the-story-of-getting-twitter-data-and-its-missing-middle/
Other
9 stars 7 forks source link

Explain rate limiting super clearly #23

Closed frabcus closed 10 years ago

frabcus commented 10 years ago

The constraints, and their current position and options.

frabcus commented 10 years ago

Things to say:

180 queries per 15 minute window (https://dev.twitter.com/docs/rate-limiting/1.1/limits) Each one can return 100 tweets https://dev.twitter.com/docs/api/1.1/get/search/tweets

Scheduled once per hour to use up those 180 queries once - so in theory four tools should go. If there aren't many tweets will go slower.

Click diagnostics button to see more info about current rate limit.

frabcus commented 10 years ago

Check your list of applications here - some may be using up your limit, revoke them:

https://twitter.com/settings/applications

frabcus commented 10 years ago

Explain how backlog getting works - it gets 7 (or is 14?) days worth intermingled with recent stuff. Then if scheudled keeps getting recent stuff based on twitter ids (check).

frabcus commented 10 years ago

Retweet counts are pretty useless.

frabcus commented 10 years ago

There are now links to https://github.com/frabcus/twitter-search-tool/wiki/How-does-this-tool-work%3F-What-are-the-limits%3F

Need to check all of above is covered there and clear.

marks commented 10 years ago

@frabcus - Just curious - is the Twitter Streamiing API not used because it would be costlier in terms of computing resources? I am pretty sure it would alleviate most of the rate concerns.

frabcus commented 10 years ago

There's no particular reason!

We started with the tool to get Twitter followers, so it was easiest to make the search tool this way.

It has the advantage that it gets you some old Tweets (up to two weeks) immediately. And it is simpler in terms of resource use (to do with the architecture of the platform).

You're right that the streaming API could be used to make it more powerful.

Francis

On Tue, Jan 28, 2014 at 07:50:46AM -0800, Mark Silverberg wrote:

@frabcus - Just curious - is the Twitter Streamiing API not used because it would be costlier in terms of computing resources? I am pretty sure it would alleviate most of the rate concerns.

Reply to this email directly or view it on GitHub: https://github.com/frabcus/twitter-search-tool/issues/23#issuecomment-33490908

Do you have an awesome idea you never quite manage to do? http://www.awesomefoundation.org/en/chapters/liverpool/