Twitter post overload

[xurizaemon]

Currently Bolts is having an issue retrieving the Twitter feed.

==> /var/log/supervisor/bolts-stdout.log <==
[Thu Apr 27 2017 09:24:46 GMT+1200 (NZST)] INFO Logging in...
[Thu Apr 27 2017 09:24:47 GMT+1200 (NZST)] INFO Websocket URL: wss://chat.civicrm.org:443/api/v3/users/websocket
[Thu Apr 27 2017 09:24:47 GMT+1200 (NZST)] INFO Logged in as user "bolts" but not connected yet.
[Thu Apr 27 2017 09:24:48 GMT+1200 (NZST)] INFO Connecting...
[Thu Apr 27 2017 09:24:49 GMT+1200 (NZST)] INFO Connected to Mattermost.
[Thu Apr 27 2017 09:24:49 GMT+1200 (NZST)] INFO Starting pinger...
[Thu Apr 27 2017 09:24:49 GMT+1200 (NZST)] INFO Mattermost server: 3.6.0.3.6.2.4bf0296a7f517c1348c1e5740aae75e6
[Thu Apr 27 2017 09:24:49 GMT+1200 (NZST)] INFO /path/to/bolts/node_modules/hubot-rss-poller2/lib/poller-script.js is using deprecated documentation syntax
[Thu Apr 27 2017 09:24:50 GMT+1200 (NZST)] WARNING Anchors don't work well with respond, perhaps you want to use 'hear'
[Thu Apr 27 2017 09:24:50 GMT+1200 (NZST)] WARNING The regex in question was /^factoids?/i
[Thu Apr 27 2017 09:24:51 GMT+1200 (NZST)] INFO hubot-redis-brain: Using default redis on localhost:6379
[Thu Apr 27 2017 09:24:51 GMT+1200 (NZST)] INFO Starting feed poller for Stack Exchange.
[Thu Apr 27 2017 09:24:51 GMT+1200 (NZST)] INFO Starting feed poller for CiviCRM Blog.
[Thu Apr 27 2017 09:24:51 GMT+1200 (NZST)] INFO hubot-redis-brain: Data for hubot brain retrieved from Redis
[Thu Apr 27 2017 09:24:51 GMT+1200 (NZST)] INFO Brain loaded
[Thu Apr 27 2017 09:24:51 GMT+1200 (NZST)] NOTICE searching twits
Error getting tweets: Error: Rate limit exceeded
[Thu Apr 27 2017 09:24:51 GMT+1200 (NZST)] INFO Brain loaded
[Thu Apr 27 2017 09:24:51 GMT+1200 (NZST)] NOTICE searching twits
[Thu Apr 27 2017 09:24:51 GMT+1200 (NZST)] INFO Brain loaded
[Thu Apr 27 2017 09:24:51 GMT+1200 (NZST)] NOTICE searching twits
[Thu Apr 27 2017 09:24:52 GMT+1200 (NZST)] INFO Brain loaded
[Thu Apr 27 2017 09:24:52 GMT+1200 (NZST)] NOTICE searching twits
[Thu Apr 27 2017 09:24:52 GMT+1200 (NZST)] INFO Brain loaded
[Thu Apr 27 2017 09:24:52 GMT+1200 (NZST)] NOTICE searching twits
Error getting tweets: Error: Rate limit exceeded
Error getting tweets: Error: Rate limit exceeded
Error getting tweets: Error: Rate limit exceeded
Error getting tweets: Error: Rate limit exceeded

My first though was that the setTimeout in hubot-twitter-mentions was being called for each tweet, leading to multiple timeouts per each Tweet retrieved. But that doesn't seem to be it.

The multiple "brain loaded" (which is from the MM adapter) suggests that some plugin or the MM adapter is triggering multiple robots to be created ... IDK, can't tell yet.

[joemcl]

Does this use the Twitter API to fetch tweets and is "Error getting tweets: Error: Rate limit exceeded" caused by a recent change in the Twitter API rate limit? Taking a look at this - https://dev.twitter.com/rest/public/rate-limiting

[joemcl]

Seeing this: "When an application exceeds the rate limit for a given API endpoint, the API will return a HTTP 429 “Too Many Requests” response code, and the following error will be returned in the response body: { "errors": [ { "code": 88, "message": "Rate limit exceeded" } ] } "

[xurizaemon]

I think we'd be under the API limit generally, this is a 2-minute poll delay. But when hubot starts making that callback repeatedly, it goes over the limit.

OHHHH. OK.

https://github.com/xurizaemon/hubot-twitter-mention/blob/master/src/twitter-mention.coffee#L78

This line hooks into the robot.brain 'loaded' event, which which would make sense if it meant "when brain is loaded", but it means "when data is loaded from brain". Great! Joe you helped me see it. We can fix this 💃

[joemcl]

cool, glad to help.

[joemcl]

Yo. Think we can fix this in time for #CiviCon #StLouis :) ? Maybe before?