Too many reconnects causing user level 420 rate limit responses

[chris-piekarski]

When running the the sample example from the repo with a common filter tag like '#2' causes the em-twitter gem to reconnect to the filter streaming api method... which causes twitter to send the following:

{"disconnect":{"code":7,"stream_name":"","reason":"admin logout"}}

The client then reconnects and repeats the process. Doing this for just a little bit will cause Twitter to rate limit the user (respond with 420 code). The streaming api is intended to be a long-lived connection and doesn't require multiple reconnects.

Are any other em-twitter gem users seeing this issue?

[stve]

em-twitter absolutely supports long-lived connections. It also includes a reconnect implementation for backing off reconnects according to Twitter's reconnection guidelines.

Are you using on_reconnect to see how often you are being reconnected?

Do you experience the same issue using a less common filter tag?

I'll do some testing to verify this myself, but I wonder if you are falling behind and being disconnected by Twitter. Stall warnings can be used to inform you of this but there have been issues with enabling them (see #9). I'm not certain that's the issue, but call it a hunch.

[JadedEvan]

Yes, another consumer of the em-twtitter gem (through [tweetstream][(https://github.com/intridea/tweetstream)) experiencing these issues. Here's some example output from logfile showing multiple successive reconnect attempts in a very short period of time. Note that these all occurred within 8 seconds of each other

[2012-10-10T16:14:09.455105 #23557]  WARN -- : Reconnecting to Streaming API. Timeout: 0.25, retries: 0
[2012-10-10T16:14:10.380448 #23557]  WARN -- : Reconnecting to Streaming API. Timeout: 0.25, retries: 0
[2012-10-10T16:14:11.491585 #23557]  WARN -- : Reconnecting to Streaming API. Timeout: 0.25, retries: 0
[2012-10-10T16:14:12.428968 #23557]  WARN -- : Reconnecting to Streaming API. Timeout: 0.25, retries: 0
[2012-10-10T16:14:13.505714 #23557]  WARN -- : Reconnecting to Streaming API. Timeout: 0.25, retries: 0
[2012-10-10T16:14:14.601088 #23557]  WARN -- : Reconnecting to Streaming API. Timeout: 0.25, retries: 0
[2012-10-10T16:14:15.726170 #23557]  WARN -- : Reconnecting to Streaming API. Timeout: 0.25, retries: 0
[2012-10-10T16:14:16.655270 #23557]  WARN -- : Reconnecting to Streaming API. Timeout: 0.25, retries: 0
[2012-10-10T16:14:17.766856 #23557]  WARN -- : Reconnecting to Streaming API. Timeout: 0.25, retries: 0
[2012-10-10T16:14:17.959287 #23557]  WARN -- : Reconnecting to Streaming API. Timeout: 0.25, retries: 0

[chris-piekarski]

Note that these all occurred within 8 seconds of each other

I was seeing reconnect messages ~10 seconds apart. So, same ballpark at least.

[stve]

Ok, the falling behind theory doesn't really carry weight given the close proximity of time in which these are occurring. I'm going to leave a stream running overnight to see if I can gather some info. I'm using this script: https://github.com/spagalloco/tweetstream-test/blob/master/em_twitter_test.rb

Anything else you could provide to help me track this down would be greatly appreciated. Thanks guys.

[stve]

Its nearly 8 hours later and my stream is still running without a single disconnect. I was hoping I'd at least experience some of the issues that you are to help me figure out why it's reconnecting.

[chris-piekarski]

Hey Steve, I will look into this again today. So we are consistent what gem version of em-twitter and tweetstream are you using?

[stve]

Hi Chris, sounds good. My test last night was just using EM-twitter 0.2.1. There shouldn't be any difference if streaming via TweetStream, but I'll run the same stream test using 2.3.0 and we'll see what happens. Thanks.

[JadedEvan]

Hey gents,

Sorry to have raised this as an issue so early. Turns out that I had some smelly code happening in my Daemon/EventMachine instance that was causing the reconnects to occur. I was seeing this because I did not understand that the TweetStream::Client.follow method invoked its own EventMachine. I was then running this inside another EventMachine loop attempting to add a EventMachine::PeriodicTimer, hence the issue. The problem that I posted about above has been solved.

This however did raise another issue, in that I am seeing reconnects occur with a high volume of streaming traffic. I'm working with about 9,000+ twitter users. Somewhere between the 4k and 6k number I start seeing my reconnect notices (although they follow a noticeably more expected pattern). Any thoughts on this?

[chris-piekarski]

Hey guys, I retried running the em-twitter/examples/stream.rb script all night against version 1 and 1.1 of the filter api and I didn't see any disconnects (unlike last time). Seemed pretty solid this time around. Might just be a red herring issue.

[stve]

thanks Chris, keep an eye on things and please re-open if this pop up again.

[JadedEvan]

Thanks for the updates gents. Discovered my reconnect issue was because of a follow limit imposed by twitter for the Streaming API. You need approval to follow more than 5000 user streams. Blog posts to come! On Oct 12, 2012, at 10:39 AM, Steve Agalloco wrote:

thanks Chris, keep an eye on things and please re-open if this pop up again.

— Reply to this email directly or view it on GitHub.

[scouttyg]

Wanted to comment on this issue to let anyone else who is Googling have another possible solution -- We were using em-twitter to follow a few users (much less than the 5000 filter limit twitter stops you at) using the streaming API and hit this error. After Googling around (and still being stumped) we realized we were running the script on two of our local machines at once (minimized window that was running it in the background), which since we were on the same network gave us the same IP to twitter, causing the errors.

So if you are getting repeated "{"disconnect":{"code":7,"stream_name":"","reason":"admin logout"}}" statuses in your log, make sure there are no processes either on your computer or other computers on the network running the same script at the same time you are.

[stve]

@scouttyg Thanks for adding your experience here. I will modify the README to make it clear as that appears to be an easy way to get tripped up.