Closed katanacrimson closed 11 years ago
kai-koch
commented
11 years ago Which version of Twit do you use?
Twitter says in it's specification, that no \r\n is send within a tweet, only between tweets. (Question is does twitter abide by this rule? Or is the parser of twit buggy.)
katanacrimson
commented
11 years ago @kai-koch I'm using latest in 1.0.x line via npm - npm ls says 1.0.12 apparently.
As for \r\n, that's something that we're able to set in bios it seems...looks like you can do a linebreak in there and it'll take it without reformatting into just a \n. Gimme a sec, I'll grab it out of my twitter profile to verify this.
(ripped from api.twitter, /1/users/show.json )
{"id":517142024,"id_str":"517142024","name":"Katy","screen_name":"_lithion","location":"","url":null,"description":"They say that even starlight takes time to travel to and fro.\r\n\r\nDon't rely on another's twilight; find your own way forward. \r\n\r\nShine...shine...SHINE!","protected":true,"followers_count":21,"friends_count":9,"listed_count":0,"created_at":"Wed Mar 07 02:49:36 +0000 2012","favourites_count":63,"utc_offset":-21600,"time_zone":"Central Time (US & Canada)","geo_enabled":false,"verified":false,"statuses_count":9522,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"9B1414","profile_background_image_url":"http:\/\/a0.twimg.com\/images\/themes\/theme14\/bg.gif","profile_background_image_url_https":"https:\/\/si0.twimg.com\/images\/themes\/theme14\/bg.gif","profile_background_tile":true,"profile_image_url":"http:\/\/a0.twimg.com\/sticky\/default_profile_images\/default_profile_0_normal.png","profile_image_url_https":"https:\/\/si0.twimg.com\/sticky\/default_profile_images\/default_profile_0_normal.png","profile_banner_url":"https:\/\/si0.twimg.com\/profile_banners\/517142024\/1357962753","profile_link_color":"009999","profile_sidebar_border_color":"FFFFFF","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":false,"default_profile":false,"default_profile_image":true,"following":null,"follow_request_sent":null,"notifications":null}\r\n is definitely in the bio, and I remember setting that a month-ish ago.
TWITTER, WHY HAVE YOU _LIED_ TO US?! shakefist
kai-koch
commented
11 years ago The Twit parser reads \r\n as marker a) between tweets and b) as keep-alive from the twitter server. https://github.com/ttezel/twit/blob/master/lib/parser.js#L29
I check if this issue is known to twitter.
kai-koch
commented
11 years ago This is the documentation of the Stream-Api. https://dev.twitter.com/docs/streaming-apis/processing For my application connecting to the Stream-Api, I have no errors for month and the stream contains users profiles with every tweet.
Which endpoint do you connect to when you get this error? We might need a different parser for different endpoints.
katanacrimson
commented
11 years ago @kai-koch I'm seeing this occurring when I retweet stuff now too, and when someone else favorites a tweet that I've made.
I'm really thinking it's the bio, but if that were the case - aren't tweets that I'd make including the bio as well?
-- update: I just had a friend test retweet something and it puked just now:
Retweeted item would be: https://twitter.com/ClassicGameRoom/status/300723704126652416
Retweet JSON is available here:
http://api.twitter.com/1/statuses/show/300724622335283201.json
Error: Error parsing twitter reply: `{"created_at":"Sun Feb 10 21:55:11 +0000 2013","id":300724622335283201,"id_str":"300724622335283201","text":"RT @ClassicGameRoom: We had a great time recording the Sonic Shuffle review for Sega Dreamcast. Enjoy! http:\/\/t.co\/nkjSTf1Q @CGRundertow","source":"\u003ca href=\"https:\/\/sourceforge.jp\/projects\/opentween\/\" rel=\"nofollow\"\u003eOpenTween\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":145878390,"id_str":"145878390","name":"thecharrr","screen_name":"thecharrr","location":"Edge of nowhere, Illinois","url":"http:\/\/steamcommunity.com\/id\/Thecharrr","description":"The cutest woodgrain loving, old electronic hoarding, and snarky replier you'll ever meet.","protected":false,"followers_count":1227,"friends_count":1307,"listed_count":107,"created_at":"Thu May 20 02:58:10 +0000 2010","favourites_count":2045,"utc_offset":-21600,"time_zone":"Central Time (US & Canada)","geo_enabled":false,"verified":false,"statuses_count":120870,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"208995","profile_background_image_url":"http:\/\/a0.twimg.com\/profile_background_images\/608523436\/bzdet37arr9zlt2yrloh.gif","profile_background_image_url_https":"https:\/\/si0.twimg.com\/profile_background_images\/608523436\/bzdet37arr9zlt2yrloh.gif","profile_background_tile":false,"profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/3218972740\/ad65c563a21ffc1e741679252410464c_normal.png","profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/3218972740\/ad65c563a21ffc1e741679252410464c_normal.png","profile_banner_url":"https:\/\/si0.twimg.com\/profile_banners\/145878390\/1355245600","profile_link_color":"F26622","profile_sidebar_border_color":"000000","profile_sidebar_fill_color":"F29909","profile_text_color":"000000","profile_use_background_image":false,"default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweeted_status":{"created_at":"Sun Feb 10 21:51:32 +0000 2013","id":300723704126652416,"id_str":"300723704126652416","text":"We had a great time recording the Sonic Shuffle review for Sega Dreamcast. Enjoy! http:\/\/t.co\/nkjSTf1Q @CGRundertow","source":"web","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":69846835,"id_str":"69846835","name":"Classic Game Room","screen_name":"ClassicGameRoom","location":"Intergalactic Space Arcade","url":"http:\/\/www.ClassicGameRoom.com","description":"Mark Bussler, exec producer Classic Game Room video game review show. Vectrex, Atari, Sega Genesis, PS3, Xbox 360, Saturn, Dreamcast, NES, SNES, PC, 3DO reviews","protected":false,"followers_count":11644,"friends_count":405,"listed_count":223,"created_at":"Sat Aug 29 11:26:58 +0000 2009","favourites_count":3,"utc_offset":-18000,"time_zone":"Eastern Time (US & Canada)","geo_enabled":false,"verified":false,"statuses_count":2637,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"131516","profile_background_image_url":"http:\/\/a0.twimg.com\/profile_background_images\/461553615\/cgr_twitgraphicb.jpg","profile_background_image_url_https":"https:\/\/si0.twimg.com\/profile_background_images\/461553615\/cgr_twitgraphicb.jpg","profile_background_tile":true,"profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/1681716381\/Logo3_normal.jpg","profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/1681716381\/Logo3_normal.jpg","profile_link_color":"009999","profile_sidebar_border_color":"EEEEEE","profile_sidebar_fill_color":"EFEFEF","profile_text_color":"333333","profile_use_background_image":true,"default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweet_count":2,"entities":{"hashtags":[],"urls":[{"url":"http:\/\/t.co\/nkjSTf1Q","expanded_url":"http:\/\/www.youtube.com\/watch?v=D7ANVbYfb6I","display_url":"youtube.com\/watch?v=D7ANVb\u2026","indices":[82,102]}],"user_mentions":[{"screen_name":"CGRundertow","name":"CGR Undertow","id":436928822,"id_str":"436928822","indices":[103,115]}]},"favorited":false,"retweeted":true,"possibly_sensitive":false},"retweet_count":0,"entities":{"hashtags":[],"urls":[{"url":"http:\/\/t.co\/nkjSTf1Q","expanded_url":"http:\/\/www.youtube.com\/watch?v=D7ANVbYfb6I","display_url":"youtube.com\/watch?v=D7ANVb\u2026","indices":[103,123]}],"user_mentions":[{"screen_name":"ClassicGameRoom","name":"Classic Game Room","id":69846835,"id_str":"69846835","indices":[3,19]},{"screen_name":"CGRundertow","name":"CGR Undertow","id":436928822,"id_str":"436928822","indices":[124,136]}]},"favorited":false,"retweeted":false,"possibly_sensitive":true}`...cause is unknown. I see nothing outright visible that would cause the error either, no \r\n in bio, jsonlint clears it just fine.
-- edit:
I'm currently connected to the user endpoint like so: stream = t.stream('user', { with: 'followings' }) to respond to your Q.
katanacrimson
commented
11 years ago @kai-koch in case it's relevant, the code itself is open source - feel free to pry if it helps you investigate: https://github.com/damianb/rumble/blob/master/rumble.js
katanacrimson
commented
11 years ago Why is the "err" being omitted when it's throwing the error on line 40? That seems awfully wasteful of the information.
Also, I'm trying to JSON.parse these json objects manually - they're fine. I'm puzzled - going to modify the files real quick and try to reproduce this.
edit: We have something sensible now: TypeError: Cannot read property 'user' of undefined. This may lie in my own code here, but I think the try/catch in parser.js can be improved here by spitting out the error message down the road.
katanacrimson
commented
11 years ago Okay, so one bug quashed - on my end, I'm not entire sure if it's related or not though (it was affecting retweet display). I'm going to open a PR in a couple minutes to add a bit more verbose errors to parser.js.
kai-koch
commented
11 years ago The JSON is fine, strings with escaped \r\n are valid JSON, but when it contains \r\n it breaks the parser because twitter says the Stream-Api will not use it inside a twitter-object.
I am currently trying to figure out who is to blame:
So far I did not find anything that says in the user stream can have a \r\n inside the user-object. I will open an issue on the Twitter-Dev site to get clarification.
katanacrimson
commented
11 years ago @kai-koch After hacking the PR I just made into my own installation, i'm getting a bit more verbose error messages. This seems to be because of twit falling back to the "tweet" event for everything it doesn't recognize - when a "user" stream event occurs such as a follow, favorite, etc. (that isn't an actual tweet) it spits it out as a tweet - and when the user's implementation chokes, the try/catch block around the emit captures it and claims that it's an error with parsing the twitter reply.
So, this appears to be a problem with oarequest.json implementation specifically, I'd guess - we need more events available to capture the varied events in the user stream, so as to not trip the user into handling them as if they were tweets.
https://github.com/ttezel/twit/blob/master/lib/oarequest.js#L48
twit can differentiate by way of checking msg.event it appears, and then going on from there with the event details itself. It appears so far that there's a favorite and follow events (from when the current user favorites a tweet, follows someone I believe). This doesn't seem to be a parser.js problem in the end, just that the error is a bit deceptive.
Should I open a new issue for handling of new events specific to the user stream?
kai-koch
commented
11 years ago Yeah, I think this would be good.
I guess nobody used the user stream and its specific events so far.
I guess a few new if-clauses to emit the user events should do the trick and @ttezel can add them to the masterbranch.
katanacrimson
commented
11 years ago For the time being, a good way to keep these extra events out will be to just check for tweet.event and dump it if it's there.
Looks like we've got a shortcut too for figuring out what all's broadcast: https://dev.twitter.com/docs/streaming-apis/messages#User_stream_messages
holy mother of god that's quite a few user stream events...
kai-koch
commented
11 years ago It is never easy. ;-P
katanacrimson
commented
11 years ago Alright, opened up PR #35 to add support for user stream events. I'm going to close this now, shouldn't be necessary to keep open now the root problem has been discovered.
maziyarpanahi
commented
10 years ago Hi guys,
I am getting this error "Error parsing twitter reply:" sometimes in my application. I am only using Public Streaming API and I have around a billion tweets in my MongoDB. So it doesn't happen everyday but it just happens out of the blue.
Is there a way to avoid this error, it just breaks the entire app.
Thanks
error parsing twitter response. not sure why, guessing it's because of the
\r\ncontained in the bio? can't really tell. jsonlint doesn't see anything wrong with it either...This is grabbed out of stderr as well, if that helps.