Closed AshwiniDoddamaniFluke closed 7 years ago
@AshwiniDoddamaniFluke Could you show me Encoding.default_external
and Encoding.default_internal
? If you're fine setting it to 'UTF-8'
, it could be a workaround for you.
Otherwise, could you see if this would fix it?
diff --git a/lib/rest-core/middleware/json_response.rb b/lib/rest-core/middleware/json_response.rb
index 3caa7d8..bc3b446 100644
--- a/lib/rest-core/middleware/json_response.rb
+++ b/lib/rest-core/middleware/json_response.rb
@@ -30,7 +30,8 @@ module RestCore
def process response
# StackExchange returns the problematic BOM! in UTF-8, so we need to
# strip it or it would break JSON parsers (i.e. yajl-ruby and json)
- body = response[RESPONSE_BODY].to_s.sub(/\A\xEF\xBB\xBF/, '')
+ body = response[RESPONSE_BODY].to_s.force_encoding('utf-8').
+ sub(/\A\xEF\xBB\xBF/, '')
response.merge(RESPONSE_BODY => Json.decode("[#{body}]").first)
# [this].first is not needed for yajl-ruby
rescue Json.const_get(:ParseError) => error
I am thinking if there's a better way to fix it.
Made changes to json_response.rb, resolved the incompatible encoding regexp match error, but i am running to some issues which are specific to my custom code in the project. Will get back to you if this works.
Also, i am not setting Encoding.default_external and Encoding.default_internal to my knowledge. Is there a specific piece of code you want me to share? Also, i dont think we should been forcing UTF-8 every time
@godfat : Your code resolved the incompatible encoding regexp match
issue.
@AshwiniDoddamaniFluke Thanks for confirming. I am not very sure if we should apply that patch. Reasons:
It's only used for dealing with that bogus StackExchange API response in the past. I don't know if they would be adding BOM once again in UTF-8, but at least they removed that some times after I fixed it. I don't know if there are any other parties would be doing this at all. Maybe we could just remove it so that we don't need to fix it.
I am not sure why were you getting ASCII-8BIT responses? Could you share more about the encoding for the responses? For example, was the response encoded in UTF-8? Was there a special header in the response? Which HTTP client are you using? Was it httpclient? (which is the default for rest-core)
@godfat : Makes sense.
<Encoding:UTF-8>
and few are <Encoding:ASCII-8BIT>
& of course when it is ASCII-8BIT, i get the error. A little bit of background here is that i call a bunch of APIs in my initializers that make calls to various REST API services."RESPONSE_HEADERS"=>{"CONTENT_TYPE"=>"application/json", "DATE"=>"Fri, 12 May 2017 17:46:50 GMT", "SERVER"=>"Apache-Coyote/1.1", "CONTENT_LENGTH"=>"362", "CONNECTION"=>"keep-alive"}
@AshwiniDoddamaniFluke I just did some tests, and it looks like httpclient is trying to be smart here. If there's Content-Type: charset=utf-8
then it would set the encoding to be UTF-8, or whatever specified in charset, otherwise ASCII-8BIT. This makes perfect sense. I was just not aware of that.
So to perfectly solve this, we should probably use whatever encoding set by httpclient. And since we're going to detect the encoding anyway, we could just make it that only strip BOM whenever it's UTF-8 (not sure about UTF-16), and leave the else intact. On the other hand, I don't know if the JSON parser could support various encoding though... I am not sure if anyone would use non-unicode for JSON either, which would be asking for troubles.
I'll push a fix shortly.
Thanks for the information and report!
@godfat : Thank you for the knowledge sharing. Awaiting the patch.
@AshwiniDoddamaniFluke Sorry for the delay. Was distracted and occupied. I've pushed a commit: https://github.com/godfat/rest-core/commit/4123ca485ecc3b9d31749423f7039bfa1652a3a3 If it works correctly, I'll release a new version shortly.
godfat/rest-core@4123ca4 works like a charm. You're awesome! Thanks. Awaiting release..
4.0.1 should be up now, thanks for confirming!
Using - ruby - 2.4.1 rails - 5.0.0 rest-core - 4.0.0
Full Trace of the error :