search

sferik / twitter-ruby

A Ruby interface to the Twitter API.
http://www.rubydoc.info/gems/twitter
MIT License
4.58k stars 1.31k forks source link

JSON::ParserError: 757: unexpected token #641

Closed stillwyw closed 1 year ago

stillwyw commented 9 years ago

I'm trying with twitter streaming api in a rake script. and sometime on some tweet, i got this error. I believe i wasn't having this issue with rails 3.2.14 but I can't be sure of that. If I try parsing these JSON in rails console or IRB it will pass.

rake aborted!
JSON::ParserError: 757: unexpected token at '{"created_at":"Mon Jan 05 23:19:00 +0000 2015","id":552242891200815104,"id_str":"552242891200815104","text":"mente aventureira, alma inquieta, ando louco por ai \u00e9 isso que me resta...","source":"\u003ca href=\"http:\/\/twitter.com\" rel=\"nofollow\"\u003eTwitter Web Client\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":1125365822,"id_str":"1125365822","name":"Cain Oliveira","screen_name":"cainoliveiraa","location":"snap:cainoliveiraa","url":"http:\/\/Instagram.com\/cainoliveiraa","description":"minha mente \u00e9 uma maldade, voc\u00eas n\u00e3o sabem o quanto pesa !!","protected":false,"verified":false,"followers_count":802,"friends_count":430,"listed_count":1,"favourites_count":3970,"statuses_count":46338,"created_at":"Sun Jan 27 15:30:36 +0000 2013","utc_offset":-7200,"time_zone":"Brasilia","geo_enabled":true,"lang":"pt","contributors_enabled":false,"is_translator":false,"profile_background_color":"C0DEED","profile_background_image_url":"http:\/\/pbs.twimg.com\/profile_background_images\/550113929469255680\/j8S01MQj.jpeg","profile_background_image_url_https":"https:\/\/pbs.twimg.com\/profile_background_images\/550113929469255680\/j8S01MQj.jpeg","profile_background_tile":true,"profile_link_color":"0084B4","profile_sidebar_border_color":"FFFFFF","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/547782445777846273\/9bQzqKar_normal.jpeg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/547782445777846273\/9bQzqKar_normal.jpeg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/1125365822\/1415314239","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":{"id":"017453ae077eafd3","url":"https:\/\/api.twitter.com\/1.1\/geo\/id\/017453ae077eafd3.json","place_type":"city","name":"Gold Coast","full_name":"Gold Coast, Queensland","country_code":"AU","country":"Australia","bounding_box":{"type":"Polygon","coordinates":[[[153.186551712,-28.2003171475],[153.186551712,-27.708126368],[153.552171232,-27.708126368],[153.552171232,-28.2003171475]]]},"attributes":{}},"contributors":null,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[],"trends":[],"urls":[],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"medium","lang":"pt","timestamp_ms":"1420499940096"}'

rails 4.1.6 ruby 2.1.1 twitter gem 5.13

Thank you :)

stve commented 9 years ago

@stillwyw can you post some of your code so that I can try to reproduce? Parse errors are not uncommon based on what we've seen with Tweetstream tweetstream/tweetstream#166

stillwyw commented 9 years ago

sure this is the task:

task :public_stream => :environment do
    san_francisco = "-122.75,36.8,-121.75,37.8"
    australia     = "112,-44,154,-9"
    puts "starting ....."
      @client.filter(locations: australia) do |tweet|
        if tweet.geo?
          if tweet.user
            #saving user normal active record saving codes
            user = User.find_or_create_by_user_obj(tweet.user)
            #saving tweet .....
            Tweet.create_by_tweet_obj_and_user_id(tweet,user.id)
          end
          puts "."
        end
      end
  end

and this is the full error message:

rake aborted!
JSON::ParserError: 757: unexpected token at '{"created_at":"Mon Jan 05 23:19:00 +0000 2015","id":552242891200815104,"id_str":"552242891200815104","text":"mente aventureira, alma inquieta, ando louco por ai \u00e9 isso que me resta...","source":"\u003ca href=\"http:\/\/twitter.com\" rel=\"nofollow\"\u003eTwitter Web Client\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":1125365822,"id_str":"1125365822","name":"Cain Oliveira","screen_name":"cainoliveiraa","location":"snap:cainoliveiraa","url":"http:\/\/Instagram.com\/cainoliveiraa","description":"minha mente \u00e9 uma maldade, voc\u00eas n\u00e3o sabem o quanto pesa !!","protected":false,"verified":false,"followers_count":802,"friends_count":430,"listed_count":1,"favourites_count":3970,"statuses_count":46338,"created_at":"Sun Jan 27 15:30:36 +0000 2013","utc_offset":-7200,"time_zone":"Brasilia","geo_enabled":true,"lang":"pt","contributors_enabled":false,"is_translator":false,"profile_background_color":"C0DEED","profile_background_image_url":"http:\/\/pbs.twimg.com\/profile_background_images\/550113929469255680\/j8S01MQj.jpeg","profile_background_image_url_https":"https:\/\/pbs.twimg.com\/profile_background_images\/550113929469255680\/j8S01MQj.jpeg","profile_background_tile":true,"profile_link_color":"0084B4","profile_sidebar_border_color":"FFFFFF","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/547782445777846273\/9bQzqKar_normal.jpeg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/547782445777846273\/9bQzqKar_normal.jpeg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/1125365822\/1415314239","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":{"id":"017453ae077eafd3","url":"https:\/\/api.twitter.com\/1.1\/geo\/id\/017453ae077eafd3.json","place_type":"city","name":"Gold Coast","full_name":"Gold Coast, Queensland","country_code":"AU","country":"Australia","bounding_box":{"type":"Polygon","coordinates":[[[153.186551712,-28.2003171475],[153.186551712,-27.708126368],[153.552171232,-27.708126368],[153.552171232,-28.2003171475]]]},"attributes":{}},"contributors":null,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[],"trends":[],"urls":[],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"medium","lang":"pt","timestamp_ms":"1420499940096"}'
/Users/yunwei/.rvm/gems/ruby-2.1.1/gems/twitter-5.13.0/lib/twitter/streaming/response.rb:27:in `block in on_body'
/Users/yunwei/.rvm/gems/ruby-2.1.1/gems/twitter-5.13.0/lib/twitter/streaming/response.rb:25:in `each'
/Users/yunwei/.rvm/gems/ruby-2.1.1/gems/twitter-5.13.0/lib/twitter/streaming/response.rb:25:in `on_body'
/Users/yunwei/.rvm/gems/ruby-2.1.1/gems/twitter-5.13.0/lib/twitter/streaming/response.rb:16:in `<<'
/Users/yunwei/.rvm/gems/ruby-2.1.1/gems/twitter-5.13.0/lib/twitter/streaming/response.rb:16:in `<<'
/Users/yunwei/.rvm/gems/ruby-2.1.1/gems/twitter-5.13.0/lib/twitter/streaming/connection.rb:22:in `stream'
/Users/yunwei/.rvm/gems/ruby-2.1.1/gems/twitter-5.13.0/lib/twitter/streaming/client.rb:118:in `request'
/Users/yunwei/.rvm/gems/ruby-2.1.1/gems/twitter-5.13.0/lib/twitter/streaming/client.rb:37:in `filter'
/Users/yunwei/projects/aussie_tweets/lib/tasks/tweet.rake:14:in `block (2 levels) in <top (required)>'
/Users/yunwei/.rvm/gems/ruby-2.1.1/bin/ruby_executable_hooks:15:in `eval'
/Users/yunwei/.rvm/gems/ruby-2.1.1/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => tweet:public_stream
(See full trace by running task with --trace)
KLFrost commented 9 years ago

Here is an example tweet which produces this error, probably related to the presence of problematic (?) characters in the text?

foo = "{\"created_at\":\"Mon Mar 23 19:51:20 +0000 2015\",\"id\":580094497834565632,\"id_str\":\"580094497834565632\",\"text\":\"RT @OfficialBBMAs: This girl will never go out of #STYLE... @taylorswift13 remains at the No. 1 spot on @billboard's #Artist100 chart\xED\xA0\xBD\xED\xB2\x9B\xED\xA0\xBD\xED\xB2\x9B\xED\xA0\xBD\xED\xB2\x9B ht…\",\"source\":\"<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>\",\"truncated\":false,\"in_reply_to_status_id\":null,\"in_reply_to_status_id_str\":null,\"in_reply_to_user_id\":null,\"in_reply_to_user_id_str\":null,\"in_reply_to_screen_name\":null,\"user\":{\"id\":2343997024,\"id_str\":\"2343997024\",\"name\":\"paige ellswick\",\"screen_name\":\"mpaige_8214\",\"location\":\"\",\"url\":null,\"description\":\"20. Married to my bestfriend, Levi. 8/2/14. Rest in peace my fur baby, Diesel. Ohio proud.\",\"protected\":false,\"verified\":false,\"followers_count\":367,\"friends_count\":260,\"listed_count\":1,\"favourites_count\":1585,\"statuses_count\":4676,\"created_at\":\"Fri Feb 14 23:33:41 +0000 2014\",\"utc_offset\":null,\"time_zone\":null,\"geo_enabled\":true,\"lang\":\"en\",\"contributors_enabled\":false,\"is_translator\":false,\"profile_background_color\":\"C0DEED\",\"profile_background_image_url\":\"http://abs.twimg.com/images/themes/theme1/bg.png\",\"profile_background_image_url_https\":\"https://abs.twimg.com/images/themes/theme1/bg.png\",\"profile_background_tile\":false,\"profile_link_color\":\"0084B4\",\"profile_sidebar_border_color\":\"C0DEED\",\"profile_sidebar_fill_color\":\"DDEEF6\",\"profile_text_color\":\"333333\",\"profile_use_background_image\":true,\"profile_image_url\":\"http://pbs.twimg.com/profile_images/576974232502067200/6FyAg-u__normal.jpeg\",\"profile_image_url_https\":\"https://pbs.twimg.com/profile_images/576974232502067200/6FyAg-u__normal.jpeg\",\"profile_banner_url\":\"https://pbs.twimg.com/profile_banners/2343997024/1418345210\",\"default_profile\":true,\"default_profile_image\":false,\"following\":null,\"follow_request_sent\":null,\"notifications\":null},\"geo\":null,\"coordinates\":null,\"place\":null,\"contributors\":null,\"retweeted_status\":{\"created_at\":\"Mon Mar 23 17:15:24 +0000 2015\",\"id\":580055254529511424,\"id_str\":\"580055254529511424\",\"text\":\"This girl will never go out of #STYLE... @taylorswift13 remains at the No. 1 spot on @billboard's #Artist100 chart\xED\xA0\xBD\xED\xB2\x9B\xED\xA0\xBD\xED\xB2\x9B\xED\xA0\xBD\xED\xB2\x9B http://t.co/b7d8wIno83\",\"source\":\"<a href=\"https://about.twitter.com/products/tweetdeck\" rel=\"nofollow\">TweetDeck</a>\",\"truncated\":false,\"in_reply_to_status_id\":null,\"in_reply_to_status_id_str\":null,\"in_reply_to_user_id\":null,\"in_reply_to_user_id_str\":null,\"in_reply_to_screen_name\":null,\"user\":{\"id\":278662460,\"id_str\":\"278662460\",\"name\":\"BillboardMusicAwards\",\"screen_name\":\"OfficialBBMAs\",\"location\":\"\",\"url\":\"http://www.billboardmusicawards.com/\",\"description\":\"The Billboard Music Awards honors some of the hottest names in music today. FB: https://www.facebook.com/TheBBMAs INSTA: https://instagram.com/officialbbmas/\",\"protected\":false,\"verified\":true,\"followers_count\":102381,\"friends_count\":923,\"listed_count\":265,\"favourites_count\":1008,\"statuses_count\":1630,\"created_at\":\"Thu Apr 07 18:26:22 +0000 2011\",\"utc_offset\":-25200,\"time_zone\":\"Pacific Time (US & Canada)\",\"geo_enabled\":false,\"lang\":\"en\",\"contributors_enabled\":false,\"is_translator\":false,\"profile_background_color\":\"FFFFFF\",\"profile_background_image_url\":\"http://pbs.twimg.com/profile_background_images/450729849036894208/GDg4Yfzh.jpeg\",\"profile_background_image_url_https\":\"https://pbs.twimg.com/profile_background_images/450729849036894208/GDg4Yfzh.jpeg\",\"profile_background_tile\":false,\"profile_link_color\":\"000000\",\"profile_sidebar_border_color\":\"FFFFFF\",\"profile_sidebar_fill_color\":\"DDEEF6\",\"profile_text_color\":\"333333\",\"profile_use_background_image\":false,\"profile_image_url\":\"http://pbs.twimg.com/profile_images/578602815368486912/c-rKKI6D_normal.jpeg\",\"profile_image_url_https\":\"https://pbs.twimg.com/profile_images/578602815368486912/c-rKKI6D_normal.jpeg\",\"profile_banner_url\":\"https://pbs.twimg.com/profile_banners/278662460/1400472999\",\"default_profile\":false,\"default_profile_image\":false,\"following\":null,\"follow_request_sent\":null,\"notifications\":null},\"geo\":null,\"coordinates\":null,\"place\":null,\"contributors\":null,\"retweet_count\":2870,\"favorite_count\":4828,\"entities\":{\"hashtags\":[{\"text\":\"STYLE\",\"indices\":[31,37]},{\"text\":\"Artist100\",\"indices\":[98,108]}],\"trends\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"taylorswift13\",\"name\":\"Taylor Swift\",\"id\":17919972,\"id_str\":\"17919972\",\"indices\":[41,55]},{\"screen_name\":\"billboard\",\"name\":\"billboard\",\"id\":9695312,\"id_str\":\"9695312\",\"indices\":[85,95]}],\"symbols\":[],\"media\":[{\"id\":580055170807001088,\"id_str\":\"580055170807001088\",\"indices\":[118,140],\"media_url\":\"http://pbs.twimg.com/media/CAzFGsYUcAAyJwS.png\",\"media_url_https\":\"https://pbs.twimg.com/media/CAzFGsYUcAAyJwS.png\",\"url\":\"http://t.co/b7d8wIno83\",\"display_url\":\"pic.twitter.com/b7d8wIno83\",\"expanded_url\":\"http://twitter.com/OfficialBBMAs/status/580055254529511424/photo/1\",\"type\":\"photo\",\"sizes\":{\"medium\":{\"w\":599,\"h\":512,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"small\":{\"w\":340,\"h\":290,\"resize\":\"fit\"},\"large\":{\"w\":599,\"h\":512,\"resize\":\"fit\"}}}]},\"extended_entities\":{\"media\":[{\"id\":580055170807001088,\"id_str\":\"580055170807001088\",\"indices\":[118,140],\"media_url\":\"http://pbs.twimg.com/media/CAzFGsYUcAAyJwS.png\",\"media_url_https\":\"https://pbs.twimg.com/media/CAzFGsYUcAAyJwS.png\",\"url\":\"http://t.co/b7d8wIno83\",\"display_url\":\"pic.twitter.com/b7d8wIno83\",\"expanded_url\":\"http://twitter.com/OfficialBBMAs/status/580055254529511424/photo/1\",\"type\":\"photo\",\"sizes\":{\"medium\":{\"w\":599,\"h\":512,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"small\":{\"w\":340,\"h\":290,\"resize\":\"fit\"},\"large\":{\"w\":599,\"h\":512,\"resize\":\"fit\"}}}]},\"favorited\":false,\"retweeted\":false,\"possibly_sensitive\":false,\"filter_level\":\"low\",\"lang\":\"en\"},\"retweet_count\":0,\"favorite_count\":0,\"entities\":{\"hashtags\":[{\"text\":\"STYLE\",\"indices\":[50,56]},{\"text\":\"Artist100\",\"indices\":[117,127]}],\"trends\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"OfficialBBMAs\",\"name\":\"BillboardMusicAwards\",\"id\":278662460,\"id_str\":\"278662460\",\"indices\":[3,17]},{\"screen_name\":\"taylorswift13\",\"name\":\"Taylor Swift\",\"id\":17919972,\"id_str\":\"17919972\",\"indices\":[60,74]},{\"screen_name\":\"billboard\",\"name\":\"billboard\",\"id\":9695312,\"id_str\":\"9695312\",\"indices\":[104,114]}],\"symbols\":[],\"media\":[{\"id\":580055170807001088,\"id_str\":\"580055170807001088\",\"indices\":[139,140],\"media_url\":\"http://pbs.twimg.com/media/CAzFGsYUcAAyJwS.png\",\"media_url_https\":\"https://pbs.twimg.com/media/CAzFGsYUcAAyJwS.png\",\"url\":\"http://t.co/b7d8wIno83\",\"display_url\":\"pic.twitter.com/b7d8wIno83\",\"expanded_url\":\"http://twitter.com/OfficialBBMAs/status/580055254529511424/photo/1\",\"type\":\"photo\",\"sizes\":{\"medium\":{\"w\":599,\"h\":512,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"small\":{\"w\":340,\"h\":290,\"resize\":\"fit\"},\"large\":{\"w\":599,\"h\":512,\"resize\":\"fit\"}},\"source_status_id\":580055254529511424,\"source_status_id_str\":\"580055254529511424\"}]},\"extended_entities\":{\"media\":[{\"id\":580055170807001088,\"id_str\":\"580055170807001088\",\"indices\":[139,140],\"media_url\":\"http://pbs.twimg.com/media/CAzFGsYUcAAyJwS.png\",\"media_url_https\":\"https://pbs.twimg.com/media/CAzFGsYUcAAyJwS.png\",\"url\":\"http://t.co/b7d8wIno83\",\"display_url\":\"pic.twitter.com/b7d8wIno83\",\"expanded_url\":\"http://twitter.com/OfficialBBMAs/status/580055254529511424/photo/1\",\"type\":\"photo\",\"sizes\":{\"medium\":{\"w\":599,\"h\":512,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"small\":{\"w\":340,\"h\":290,\"resize\":\"fit\"},\"large\":{\"w\":599,\"h\":512,\"resize\":\"fit\"}},\"source_status_id\":580055254529511424,\"source_status_id_str\":\"580055254529511424\"}]},\"favorited\":false,\"retweeted\":false,\"possibly_sensitive\":false,\"filter_level\":\"low\",\"lang\":\"en\",\"timestamp_ms\":\"1427140280575\"}"
JSON.parse(foo)
tibbon commented 7 years ago

I'm having a similar but different issue here: https://github.com/sferik/twitter/issues/849

I've written a failing test on https://github.com/sferik/twitter/pull/850