rack fails on %-encoding for certain tweets

[strworkstation]

Raising this issue despite the failure being outside of the Datasift code as it may render the push API unusable for Rails developers.

• This is the known Rack issue: https://github.com/rack/rack/issues/337
• Background: a specific Tweet interaction (https://twitter.com/MonthlyFailz/status/247787409662689280) was triggering an exception in our push endpoint, but it happened in Rack, rather than our app code. It turns out that when Rack passes the POST body to Rails, it chokes when un-encoding the POST body, but only for this tweet interaction.
• To test possible solutions, we monkey-patched the URI module in our test environment. We confirmed that this monkey-patch fixes the issue encountered with the datasift push: https://raw.github.com/gist/3130349/894358c7a34a1c2b62568577e942643e6fa85dd8/URI-monkey-patch.rb
• we are using version 2.0.1 of the datasift-ruby gem
• backtrace of the error:

[2012-09-18 14:48:21] ERROR ArgumentError: invalid %-encoding (\"http://www.echofon.com/\" rel=\"nofollow\">Echofon</a>","created_at":"Tue, 18 Sep 2012 21:42:56 +0000"}}},{"interaction":{"schema":{"version":3},"source":"Google","author":{"username":"HarajukuNoMatie","name":"\uFF4D\uFF41\uFF54\uFF49\uFF45/ \u30DE\u30C1\u30A8 \u0CA5\u2323\u0CA5","id":173483860,"avatar":"http://a0.twimg.com/profile_images/2546758795/8g7tb810uy2rn7tjmiiv_normal.gif","link":"http://twitter.com/HarajukuNoMatie"},"type":"twitter","created_at":"Tue, 18 Sep 2012 21:20:42 +0000","content":"Gostei de um v\u00EDdeo do @YouTube http://t.co/F2rOv0Av Gotye - Somebody That I Used To Know (feat. Kimbra) -","id":"1e201d6b375ba100e0749d0b9256a42c","link":"http://twitter.com/HarajukuNoMatie/statuses/248169701144924160","tags":["2672"]},"language":{"tag":"pt","confidence":100},"links":{"created_at":["Tue, 05 Jul 2011 22:41:00 +0000"],"retweet_count":[61908],"title":["YouTube - Gotye feat. Kimbra- Somebody That I Used To Know- official film clip (HD)"],"url":["http://www.youtube.com/watch?v=8UVNT4wvIGY"]},"salience":{"content":{"sentiment":-2}},"twitter":{"created_at":"Tue, 18 Sep 2012 21:20:42 +0000","domains":["youtu.be"],"id":"248169701144924160","links":["http://youtu.be/8UVNT4wvIGY?a"],"mention_ids":["10228272"],"mentions":["YouTube"],"source":"<a href=\"http://www.google.com/\" rel=\"nofollow\">Google</a>","text":"Gostei de um v\u00EDdeo do @YouTube http://t.co/F2rOv0Av Gotye - Somebody That I Used To Know (feat. Kimbra) -","user":{"name":"\uFF4D\uFF41\uFF54\uFF49\uFF45/ \u30DE\u30C1\u30A8 \u0CA5\u2323\u0CA5","url":"http://akairo-no-jelly.tumblr.com/","description":" http://24.media.tumblr.com/tumblr_m1v9owC0WO1qczbido1_500.gif  (\uFF89\u25D5\u30EE\u25D5)\uFF89*:\uFF65\uFF9F\u2727 @DIVA_MATIE\r\nhttp://soundcloud.com/harajukunomatie","location":"Wonderland","statuses_count":40280,"followers_count":281,"friends_count":380,"screen_name":"HarajukuNoMatie","lang":"en","time_zone":"Santiago","utc_offset":-14400,"listed_count":6,"id":173483860,"id_str":"173483860","geo_enabled":true,"created_at":"Sun, 01 Aug 2010 16:03:40 +0000"}}},{"interaction":{"schema":{"version":3},"source":"web","author":{"username":"StrFoo","name":"Str Foo","id":326477838,"avatar":"http://a0.twimg.com/sticky/default_profile_images/default_profile_1_normal.png","link":"http://twitter.com/StrFoo"},"type":"twitter","created_at":"Tue, 18 Sep 2012 21:20:55 +0000","content":"RT @MonthlyFailz: :: Sheldon Cooper - Singer Version :: (@YouTube http://t.co/zAY1jPus)","id":"1e201d6bb355a580e0740f9d4c8c9846","link":"http://twitter.com/StrFoo/statuses/248169758632058881","tags":["2672"]},"language":{"tag":"da","confidence":100},"links":{"created_at":["Tue, 05 Jul 2011 22:41:00 +0000"],"retweet_count":[61908],"title":["YouTube - Gotye feat. Kimbra- Somebody That I Used To Know- official film clip (HD)"],"url":["http://www.youtube.com/watch?v=8UVNT4wvIGY"]},"twitter":{"id":"248169758632058881","retweet":{"text":":: Sheldon Cooper - Singer Version :: (@YouTube http://t.co/zAY1jPus)","id":"248169758632058881","user":{"name":"Str Foo","statuses_count":4,"screen_name":"StrFoo","lang":"en","id":326477838,"id_str":"326477838","created_at":"Thu, 30 Jun 2011 00:06:10 +0000"},"source":"web","count":3,"created_at":"Tue, 18 Sep 2012 21:20:55 +0000","mentions":["YouTube"],"mention_ids":["10228272"],"links":["http://youtu.be/8UVNT4wvIGY?a"],"domains":["youtu.be"]},"retweeted":{"id":"247787409662689280","user":{"name":"Monthly Failz","url":"http://www.youtube.com/MonthlyFailz","description":"1  \uFF2D O \uFF2E T H - 1  V\uFF29D - 1 0 0 %  F\uFF21\uFF29L S","statuses_count":13,"followers_count":3,"friends_count":58,"screen_name":"MonthlyFailz","lang":"en","time_zone":"Eastern Time (US )
    /Users/engineer/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/uri/common.rb:898:in `decode_www_form_component'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/utils.rb:39:in `unescape'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/utils.rb:88:in `block (2 levels) in parse_nested_query'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/utils.rb:88:in `map'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/utils.rb:88:in `block in parse_nested_query'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/utils.rb:87:in `each'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/utils.rb:87:in `parse_nested_query'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/request.rb:335:in `parse_query'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/request.rb:210:in `POST'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/methodoverride.rb:26:in `method_override'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/methodoverride.rb:14:in `call'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/airbrake-3.1.3/lib/airbrake/rack.rb:41:in `call'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/airbrake-3.1.3/lib/airbrake/user_informer.rb:12:in `call'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/content_length.rb:14:in `call'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/railties-3.2.8/lib/rails/rack/log_tailer.rb:17:in `call'
    /Users/engineer/.rvm/gems/ruby-1.9.3-p125@Creatorious/gems/rack-1.4.1/lib/rack/handler/webrick.rb:59:in `service'
    /Users/engineer/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/webrick/httpserver.rb:138:in `service'
    /Users/engineer/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/webrick/httpserver.rb:94:in `run'
    /Users/engineer/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/webrick/server.rb:191:in `block in start_thread'

[stut]

The POST data we send is not form encoded, so no form decoding should be being performed. Unfortunately we don't have any experience with Rack, but if there's a way to disable that automatic decoding then you should be doing that and handling the POST data as a JSON document.

[stut]

A quick Google found this as a possible solution: https://github.com/achiu/rack-parser