search

voloko / twitter-stream

Twitter realtime API client
MIT License
233 stars 81 forks source link

no_data_callback not being called when newlines (\n\r) are being received #38

Open mgartner opened 10 years ago

mgartner commented 10 years ago

Twice in the past month I've had my long running Twitter Stream consumer stall and not receive any tweets, without it calling the no_data_callback. I'm wondering if this could be because the Streaming API is sending whitespace (\n or \r or both) and this whitespace is actually treated as data at https://github.com/voloko/twitter-stream/blob/master/lib/twitter/json_stream.rb#L130. This line that resets @last_data_received_at doesn't seem to check whether the data is whitespace or not - and maybe that's why the no_data_callback is never called.

I guess this would be the wrong place to set @last_data_received_at anyway because it can't know that the data is whitespace without first parsing it.

I'm happy to work on a fix, but I wanted to get some feedback first on whether or not my logic seems sound.

mgartner commented 10 years ago

I thought this was definitely the issue I was having, but after deploying the fix, I experienced the same hang in the stream only a few days later. I don't know what's going on, but it seems like either Twitter is sending gibberish that isn't empty but also isn't parsed; or they are sending nothing or empty data and EventMachine's periodic timer is broken.

It's a mystery how this library has running so well for over 2 years with only a couple of freezes in that time, and now in the last two months I've had 4-5 with no obviously relevant changes to the library or my client code.

Any help in debugging this is greatly appreciated.