Still get "invalid byte sequence in UTF-8" errors with Internet Explorer

[ncri]

Tested IE 11 on Win 7

[sbleon]

What input is causing the trouble? It could be the URL or POST data.

On May 30, 2014, at 3:32 AM, Nico Ritsche notifications@github.com wrote:

Tested IE 11 on Win 7

— Reply to this email directly or view it on GitHub.

[ncri]

For example an ajax get request to this path in our rails 4 app:

?searchterm=Ausbildung%20DRK-Landesverband%20Baden-Württemberg%20e.V.&place=&distance=0&=1401440064261

But only for IE. I tried adding the snowman parameter that rails uses in forms to force IE to submit UTF-8 but that doesn't help: &_snowman=☃

[ncri]

here is the full stack trace (the error occurs when calling gsub on params[:search_term]):

/app/app/controllers/kimeta_controller.rb:17:in `gsub'
/app/app/controllers/kimeta_controller.rb:17:in `results'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/abstract_controller/base.rb:189:in `process_action'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_controller/metal/rendering.rb:10:in `process_action'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:113:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:113:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:229:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:229:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:215:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:215:in `block in halting_and_conditional'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:229:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:229:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:229:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:229:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:149:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:229:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:229:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:166:in `block in halting'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:86:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:86:in `run_callbacks'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/abstract_controller/callbacks.rb:19:in `process_action'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_controller/metal/rescue.rb:29:in `process_action'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_controller/metal/instrumentation.rb:31:in `block in process_action'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/notifications.rb:159:in `block in instrument'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/notifications.rb:159:in `instrument'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/app/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/abstract_controller/base.rb:136:in `process'
/app/vendor/bundle/ruby/2.1.0/gems/actionview-4.1.1/lib/action_view/rendering.rb:30:in `process'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_controller/metal.rb:195:in `dispatch'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_controller/metal.rb:231:in `block in action'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/routing/route_set.rb:80:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/routing/route_set.rb:80:in `dispatch'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/routing/route_set.rb:48:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/journey/router.rb:71:in `block in call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/journey/router.rb:59:in `each'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/journey/router.rb:59:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/routing/route_set.rb:676:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/omniauth-1.2.1/lib/omniauth/strategy.rb:186:in `call!'
/app/vendor/bundle/ruby/2.1.0/gems/omniauth-1.2.1/lib/omniauth/strategy.rb:164:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.7.3.204/lib/new_relic/rack/error_collector.rb:55:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.7.3.204/lib/new_relic/rack/agent_hooks.rb:32:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.7.3.204/lib/new_relic/rack/browser_monitoring.rb:27:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/jquery-fileupload-rails-0.4.1/lib/jquery/fileupload/rails/middleware.rb:14:in `_call'
/app/vendor/bundle/ruby/2.1.0/gems/jquery-fileupload-rails-0.4.1/lib/jquery/fileupload/rails/middleware.rb:10:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/warden-1.2.3/lib/warden/manager.rb:35:in `block in call'
/app/vendor/bundle/ruby/2.1.0/gems/warden-1.2.3/lib/warden/manager.rb:34:in `catch'
/app/vendor/bundle/ruby/2.1.0/gems/warden-1.2.3/lib/warden/manager.rb:34:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/etag.rb:23:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/conditionalget.rb:25:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/middleware/flash.rb:254:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:225:in `context'
/app/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:220:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/query_cache.rb:36:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:621:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/callbacks.rb:82:in `run_callbacks'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/railties-4.1.1/lib/rails/rack/logger.rb:38:in `call_app'
/app/vendor/bundle/ruby/2.1.0/gems/railties-4.1.1/lib/rails/rack/logger.rb:20:in `block in call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/tagged_logging.rb:68:in `block in tagged'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/tagged_logging.rb:26:in `tagged'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/tagged_logging.rb:68:in `tagged'
/app/vendor/bundle/ruby/2.1.0/gems/railties-4.1.1/lib/rails/rack/logger.rb:20:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/methodoverride.rb:21:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/runtime.rb:17:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/cache/strategy/local_cache_middleware.rb:26:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/heroku_rails_deflate-1.0.3/lib/heroku_rails_deflate/serve_zipped_assets.rb:58:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/deflater.rb:25:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
/app/vendor/bundle/ruby/2.1.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'
/app/vendor/bundle/ruby/2.1.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:172:in `rescue in lookup'
/app/vendor/bundle/ruby/2.1.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:168:in `lookup'
/app/vendor/bundle/ruby/2.1.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!'
/app/vendor/bundle/ruby/2.1.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/actionpack-4.1.1/lib/action_dispatch/middleware/static.rb:64:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/sendfile.rb:112:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/utf8-cleaner-0.0.9/lib/utf8-cleaner/middleware.rb:18:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/railties-4.1.1/lib/rails/engine.rb:514:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/railties-4.1.1/lib/rails/application.rb:144:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/railties-4.1.1/lib/rails/railtie.rb:194:in `public_send'
/app/vendor/bundle/ruby/2.1.0/gems/railties-4.1.1/lib/rails/railtie.rb:194:in `method_missing'
/app/vendor/bundle/ruby/2.1.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:572:in `process_client'
/app/vendor/bundle/ruby/2.1.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:666:in `worker_loop'
/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.7.3.204/lib/new_relic/agent/instrumentation/unicorn_instrumentation.rb:22:in `call'
/app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.7.3.204/lib/new_relic/agent/instrumentation/unicorn_instrumentation.rb:22:in `block (4 levels) in '
/app/vendor/bundle/ruby/2.1.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:521:in `spawn_missing_workers'
/app/vendor/bundle/ruby/2.1.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:140:in `start'
/app/vendor/bundle/ruby/2.1.0/gems/unicorn-4.8.2/bin/unicorn:126:in `'
/app/vendor/bundle/ruby/2.1.0/bin/unicorn:23:in `load'
/app/vendor/bundle/ruby/2.1.0/bin/unicorn:23:in `'

[henrik]

@ncri Is the param in the URL of the page containing the form or is it part of what the form submits? Perhaps it must be the latter: http://stackoverflow.com/questions/153527/setting-the-character-encoding-in-form-submit-for-internet-explorer

[ncri]

Well, actually I tried that already, no difference...

[bowd]

@ncri I'm getting the same thing, is there any information on this? It happens for AJAX request coming from multiple IE versions (9-11), when UTF-8 characters are included in the request. Example request:

  * URL       : /users/8387/edit?view=guide_associations&location=ירושלים,%20ישראל
  * IP address: 79.179.54.252
  * Parameters: {"view"=>"guide_associations", "location"=>"\xE9\xF8\xE5\xF9\xEC\xE9\xED, \xE9\xF9\xF8\xE0\xEC", "action"=>"edit", "controller"=>"users", "id"=>"8387"}
  * Rails root: /app
  * Timestamp : 2014-08-18 08:19:12 UTC

[ncri]

Well, I haven't found a full solution yet. I just hack around it by doing some character replacements:

# IE hack: re-encode search term if it contains invalid chars
begin
  my_string.gsub('', '')   # this raises an error for invalid strings
rescue ArgumentError
  my_string.encode!( 'UTF-8', 'Windows-1252' )
end

Unsatisfying because it doesn't always seem to work.

[valscion]

Hi, I'm having the same issue.

I tested with a request where env["QUERY_STRING"] before sanitation was H\xE4meenlinna — clearly "Hämeenlinna" encoded in Windows-1252 and not UTF-8. I then checked what I'd get as a result of running

sanitize_env(env)["QUERY_STRING"]

in debugger while I stopped at the middleware #call method — and I still got H\xE4meenlinna out, which was clearly incorrect, and it caused the request to crash and burn, raising "invalid byte sequence in UTF-8" error later on when I did any string operations on it.

I'm happy to create a test case for this scenario, if you so wish.

EDIT: I tested on IE8, IE9, IE10 and IE11 on virtual machines — all returning the same result. Having UTF-8 snowman in query parameters did not help. This request was a normal GET request.

I'm running Rails 4.2.3 and version 0.0.9 of the utf8-cleaner gem.

[sbleon]

Please do go ahead and create a test.

Since \xE4 is not valid URI encoding (aka % encoding), the behavior I'd suggest in this situation would be one of:

1. Remove the \, leaving xE4
2. Remove the entire 4 bytes of the character
3. Double the \ so that it is escaped and doesn't get decoded.

I don't want UTF8-cleaner to have the responsibility of translating other character sets (e.g. CP1252) to UTF-8, and I don't want it to facilitate the use of improper encodings in URLs. I'm leaning towards option 3.

---

Leon Miller-Out - Singlebrook Technology - 607-330-0191

On Fri, Jul 31, 2015 at 3:46 AM, Vesa Laakso notifications@github.com wrote:

Hi, I'm having the same issue.

I tested with a request where env["QUERY_STRING"] before sanitation was H\xE4meenlinna — clearly "Hämeenlinna" encoded in Windows-1252 and not UTF-8. I then checked what I'd get as a result of running

sanitize_env(env)["QUERY_STRING"]

in debugger while I stopped at the middleware #call method — and I still got H\xE4meenlinna out, which was clearly incorrect and caused, and it caused the request to crash and burn, raising "invalid byte sequence in UTF-8" error.

I'm happy to create a test case for this scenario, if you so wish.

— Reply to this email directly or view it on GitHub https://github.com/singlebrook/utf8-cleaner/issues/17#issuecomment-126596365 .

[valscion]

I'll see what I can do :)

I'm not sure double escaping the \x4 would result in good user experience. In our case, the value would also be prefilled in a search input.

Do you think it'd make sense to use the U+FFFD REPLACAMENT CHARACTER � in these cases instead? That way it might be more clear to the user that their special characters had some issues.

used to replace an incoming character whose value is unknown or unrepresentable in Unicode

[sbleon]

I'd rather change the input as little as possible, so I'm still for double-encoding. If there's a poor user experience, it's due to either

1. an archaic browser that doesn't do UTF-8
2. the developer not specifying UTF-8 encoding correctly in their head/headers

In neither of this cases should we trying to be helpful, in my opinion, but I do want to try to keep the request from crashing.

---

Leon Miller-Out - Singlebrook Technology - 607-330-0191

On Sat, Aug 1, 2015 at 3:22 AM, Vesa Laakso notifications@github.com wrote:

I'll see what I can do :)

I'm not sure double escaping the \x4 would result in good user experience. In our case, the value would also be prefilled in a search input.

Do you think it'd make sense to use the U+FFFD REPLACAMENT CHARACTER http://www.fileformat.info/info/unicode/char/0fffd/index.htm � in these cases instead? That way it might be more clear to the user that their special characters had some issues.

used to replace an incoming character whose value is unknown or unrepresentable in Unicode

— Reply to this email directly or view it on GitHub https://github.com/singlebrook/utf8-cleaner/issues/17#issuecomment-126877762 .

[valscion]

Yeah, you're right. Maybe the least surprising thing would be to remove the characters altogether? It feels like it would be what I'd expect this gem to do based on the README (emphasis mine):

Removes invalid UTF-8 characters from the environment

[sbleon]

Good point! Can we actually remove the characters safely? If they're in an invalid \xnn format, we might not know how many bytes they are supposed to be, so it might be hard to remove them correctly.

---

Leon Miller-Out - Singlebrook Technology - 607-330-0191

On Tue, Aug 4, 2015 at 6:20 AM, Vesa Laakso notifications@github.com wrote:

Yeah, you're right. Maybe the least surprising thing would be to remove the characters altogether? It feels like it would be what I'd expect this gem to do based on the README (emphasis mine):

Removes invalid UTF-8 characters from the environment

— Reply to this email directly or view it on GitHub https://github.com/singlebrook/utf8-cleaner/issues/17#issuecomment-127557707 .

[valscion]

A-ha! I'll make sure to add good test coverage to test these cases. If it turns out to be too difficult we'll at least have some code to talk about.

I'll try to get my hands dirty with the following week, feel free to ping me if there doesn't seem to be any progress.

[sbleon]

@ncri @valscion I think this may be addressed by the new version v0.1.1. It replaces Windows-encoded characters with their UTF8 equivalents. https://github.com/singlebrook/utf8-cleaner/blob/master/CHANGELOG.md

[valscion]

Wow, super! Thanks a bunch @sbleon :+1: I'll make sure to test this