search

voloko / twitter-stream

Twitter realtime API client
MIT License
233 stars 81 forks source link

input buffer full #3

Closed kamdh closed 14 years ago

kamdh commented 14 years ago

Hi, I've implemented a daemon which uses twitter-stream, but we're getting "input buffer full" messages in the logs:

[Chugger 07:05] 4564 tweets [Chugger 07:06] 4727 tweets [Chugger 07:07] Error: input buffer full [Chugger 07:07] Error: input buffer full [Chugger 07:07] Warning: reconnect in 10 seconds [Chugger 07:07] 3798 tweets [Chugger 07:08] 4626 tweets

This looks like something emanating from EM::BufferizedTokenizer. I'm not sure why this would start happening now, but twitter is adding metadata to the JSON so perhaps it's that. A possible solution: up the MAX_LINE_LENGTH (where does your current number 16*1024 come from?).

Cheers, Kameron

voloko commented 14 years ago

Fixed in 0.1.6. Upped the limit to 1MB

Twitter stream reads tweets into the string buffer and then parses received input line by line. MAX_LINE_LENGTH was set to 16kb (I think I took it from some EM example). Exceeding it means that twitter sent data in chunks larger then 16Kb or your app is too slow to process it.

kamdh commented 14 years ago

Thanks, will let you know if the problem persists. I doubt it is the app as it is quite barebones and just dumps the data into a file.