godfat / rest-core

Various rest-builder middleware for building REST clients.
Apache License 2.0
57 stars 13 forks source link

RestBuilder::Promise: ERROR: incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string) #24

Closed AshwiniDoddamaniFluke closed 7 years ago

AshwiniDoddamaniFluke commented 7 years ago

Using - ruby - 2.4.1 rails - 5.0.0 rest-core - 4.0.0

Full Trace of the error :

/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-core-4.0.0/lib/rest-core/middleware/cache.rb:71: warning: constant ::Fixnum is deprecated
RestBuilder::Promise: ERROR: incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
  from ["/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-core-4.0.0/lib/rest-core/middleware/json_response.rb:33:in `sub'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-core-4.0.0/lib/rest-core/middleware/json_response.rb:33:in `process'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-core-4.0.0/lib/rest-core/middleware/json_response.rb:26:in `block in call'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-core-4.0.0/lib/rest-core/middleware/common_logger.rb:13:in `block in call'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-core-4.0.0/lib/rest-core/middleware/cache.rb:34:in `block in app_call'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:118:in `block in resolve'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:118:in `each'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:118:in `inject'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:118:in `resolve'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:109:in `fulfilling'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:84:in `block in fulfill'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:84:in `synchronize'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:84:in `fulfill'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:136:in `protected_yield'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:49:in `block in defer'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-builder-0.9.1/lib/rest-builder/engine.rb:28:in `call'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-core-4.0.0/lib/rest-core/middleware/auth_basic.rb:10:in `call'", 
        "/Users/ashwini/no-rocket-science/sw-web/lib/fluke/token.rb:24:in `call'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-core-4.0.0/lib/rest-core/middleware/cache.rb:29:in `app_call'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-core-4.0.0/lib/rest-core/middleware/cache.rb:25:in `call'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-core-4.0.0/lib/rest-core/middleware/common_logger.rb:12:in `call'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-core-4.0.0/lib/rest-core/middleware/json_response.rb:24:in `call'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-core-4.0.0/lib/rest-core/middleware/default_site.rb:14:in `call'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rest-builder-0.9.1/lib/rest-builder/client.rb:168:in `request_full'", 
        "/Users/ashwini/no-rocket-science/sw-web/lib/fluke/client/client.rb:136:in `do_request'", 
        "/Users/ashwini/no-rocket-science/sw-web/lib/fluke/client/client.rb:81:in `get'", 
        "/Users/ashwini/no-rocket-science/sw-web/lib/fluke/client/constants_api.rb:46:in `measurement_types'", 
        "/Users/ashwini/no-rocket-science/sw-web/config/initializers/fluke_global.rb:121:in `<top (required)>'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.0/lib/active_support/dependencies.rb:287:in `load'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.0/lib/active_support/dependencies.rb:287:in `block in load'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.0/lib/active_support/dependencies.rb:259:in `load_dependency'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.0/lib/active_support/dependencies.rb:287:in `load'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/engine.rb:648:in `block in load_config_initializer'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.0/lib/active_support/notifications.rb:166:in `instrument'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/engine.rb:647:in `load_config_initializer'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/engine.rb:612:in `block (2 levels) in <class:Engine>'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/engine.rb:611:in `each'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/engine.rb:611:in `block in <class:Engine>'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/initializable.rb:30:in `instance_exec'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/initializable.rb:30:in `run'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/initializable.rb:55:in `block in run_initializers'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:228:in `block in tsort_each'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:431:in `each_strongly_connected_component_from'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:421:in `block in each_strongly_connected_component_from'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/initializable.rb:44:in `each'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/initializable.rb:44:in `tsort_each_child'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:415:in `call'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:415:in `each_strongly_connected_component_from'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:349:in `block in each_strongly_connected_component'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:347:in `each'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:347:in `call'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:347:in `each_strongly_connected_component'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:226:in `tsort_each'", 
        "/Users/ashwini/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/tsort.rb:205:in `tsort_each'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/initializable.rb:54:in `run_initializers'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/application.rb:352:in `initialize!'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/railtie.rb:193:in `public_send'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/railtie.rb:193:in `method_missing'", 
        "/Users/ashwini/no-rocket-science/sw-web/config/environment.rb:5:in `<top (required)>'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.0/lib/active_support/dependencies.rb:293:in `require'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.0/lib/active_support/dependencies.rb:293:in `block in require'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.0/lib/active_support/dependencies.rb:259:in `load_dependency'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.0/lib/active_support/dependencies.rb:293:in `require'", 
        "/Users/ashwini/no-rocket-science/sw-web/config.ru:3:in `block in <main>'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rack-2.0.1/lib/rack/builder.rb:55:in `instance_eval'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rack-2.0.1/lib/rack/builder.rb:55:in `initialize'", 
        "/Users/ashwini/no-rocket-science/sw-web/config.ru:in `new'",
        "/Users/ashwini/no-rocket-science/sw-web/config.ru:in `<main>'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rack-2.0.1/lib/rack/builder.rb:49:in `eval'",
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rack-2.0.1/lib/rack/builder.rb:49:in `new_from_string'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rack-2.0.1/lib/rack/builder.rb:40:in `parse_file'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rack-2.0.1/lib/rack/server.rb:318:in `build_app_and_options_from_config'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rack-2.0.1/lib/rack/server.rb:218:in `app'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/commands/server.rb:59:in `app'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/rack-2.0.1/lib/rack/server.rb:353:in `wrapped_app'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/commands/server.rb:124:in `log_to_stdout'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/commands/server.rb:77:in `start'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/commands/commands_tasks.rb:90:in `block in server'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/commands/commands_tasks.rb:85:in `tap'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/commands/commands_tasks.rb:85:in `server'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/commands/commands_tasks.rb:49:in `run_command!'", 
        "/Users/ashwini/.rvm/gems/ruby-2.4.1/gems/railties-5.0.0/lib/rails/commands.rb:18:in `<top (required)>'", 
        "bin/rails:9:in `require'", "bin/rails:9:in `<main>'"]
godfat commented 7 years ago

@AshwiniDoddamaniFluke Could you show me Encoding.default_external and Encoding.default_internal? If you're fine setting it to 'UTF-8', it could be a workaround for you.

Otherwise, could you see if this would fix it?

diff --git a/lib/rest-core/middleware/json_response.rb b/lib/rest-core/middleware/json_response.rb
index 3caa7d8..bc3b446 100644
--- a/lib/rest-core/middleware/json_response.rb
+++ b/lib/rest-core/middleware/json_response.rb
@@ -30,7 +30,8 @@ module RestCore
     def process response
       # StackExchange returns the problematic BOM! in UTF-8, so we need to
       # strip it or it would break JSON parsers (i.e. yajl-ruby and json)
-      body = response[RESPONSE_BODY].to_s.sub(/\A\xEF\xBB\xBF/, '')
+      body = response[RESPONSE_BODY].to_s.force_encoding('utf-8').
+        sub(/\A\xEF\xBB\xBF/, '')
       response.merge(RESPONSE_BODY => Json.decode("[#{body}]").first)
       # [this].first is not needed for yajl-ruby
     rescue Json.const_get(:ParseError) => error

I am thinking if there's a better way to fix it.

AshwiniDoddamaniFluke commented 7 years ago

Made changes to json_response.rb, resolved the incompatible encoding regexp match error, but i am running to some issues which are specific to my custom code in the project. Will get back to you if this works.

AshwiniDoddamaniFluke commented 7 years ago

Also, i am not setting Encoding.default_external and Encoding.default_internal to my knowledge. Is there a specific piece of code you want me to share? Also, i dont think we should been forcing UTF-8 every time

AshwiniDoddamaniFluke commented 7 years ago

@godfat : Your code resolved the incompatible encoding regexp match issue.

godfat commented 7 years ago

@AshwiniDoddamaniFluke Thanks for confirming. I am not very sure if we should apply that patch. Reasons:

AshwiniDoddamaniFluke commented 7 years ago

@godfat : Makes sense.

  1. Few responses are encoded in <Encoding:UTF-8> and few are <Encoding:ASCII-8BIT> & of course when it is ASCII-8BIT, i get the error. A little bit of background here is that i call a bunch of APIs in my initializers that make calls to various REST API services.
  2. Nothing special about the headers really. "RESPONSE_HEADERS"=>{"CONTENT_TYPE"=>"application/json", "DATE"=>"Fri, 12 May 2017 17:46:50 GMT", "SERVER"=>"Apache-Coyote/1.1", "CONTENT_LENGTH"=>"362", "CONNECTION"=>"keep-alive"}
  3. I am using httpclient (2.8.3)
godfat commented 7 years ago

@AshwiniDoddamaniFluke I just did some tests, and it looks like httpclient is trying to be smart here. If there's Content-Type: charset=utf-8 then it would set the encoding to be UTF-8, or whatever specified in charset, otherwise ASCII-8BIT. This makes perfect sense. I was just not aware of that.

So to perfectly solve this, we should probably use whatever encoding set by httpclient. And since we're going to detect the encoding anyway, we could just make it that only strip BOM whenever it's UTF-8 (not sure about UTF-16), and leave the else intact. On the other hand, I don't know if the JSON parser could support various encoding though... I am not sure if anyone would use non-unicode for JSON either, which would be asking for troubles.

I'll push a fix shortly.

Thanks for the information and report!

AshwiniDoddamaniFluke commented 7 years ago

@godfat : Thank you for the knowledge sharing. Awaiting the patch.

godfat commented 7 years ago

@AshwiniDoddamaniFluke Sorry for the delay. Was distracted and occupied. I've pushed a commit: https://github.com/godfat/rest-core/commit/4123ca485ecc3b9d31749423f7039bfa1652a3a3 If it works correctly, I'll release a new version shortly.

AshwiniDoddamaniFluke commented 7 years ago

godfat/rest-core@4123ca4 works like a charm. You're awesome! Thanks. Awaiting release..

godfat commented 7 years ago

4.0.1 should be up now, thanks for confirming!

https://rubygems.org/gems/rest-core/versions/4.0.1