Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT

[sainejob]

We have a penetration testing service which berates our app with traffic intended to discover some vulnerabilities. I've found that utf8-cleaner can prevent most of the exception we used to see but there is still an exception that is thrown very often.

…ctivesupport-4.0.13/lib/active_support/core_ext/uri.rb:   15:in `gsub'
…ctivesupport-4.0.13/lib/active_support/core_ext/uri.rb:   15:in `unescape'
                /usr/local/lib/ruby/2.2.0/uri/common.rb:  125:in `unescape'
…gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/uri_string.rb:   88:in `valid_uri_encoded_utf8'
…gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/uri_string.rb:   23:in `valid?'
…gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/uri_string.rb:   15:in `cleaned'
…gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/middleware.rb:   56:in `cleaned_string'
…gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/middleware.rb:   44:in `sanitize_env_rack_input'
…gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/middleware.rb:   30:in `sanitize_env'
…gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/middleware.rb:   21:in `call'

I've found this article which talks about many things including this error: http://yehudakatz.com/2010/05/05/ruby-1-9-encodings-a-primer-and-the-solution-for-rails/

I'm wondering if there is anything that utf8-cleaner could do to prevent this particular exception.

You guys have any further insight on the subject? Thanks!

[sbleon]

Hi! I’m glad that utf8-cleaner is helping you out!

Can you supply the problematic input? In descending order of preference, this could be:

1. A failing test case in the test suite
2. A curl request that triggers the error in your app

On Wed, Mar 16, 2016 at 1:55 PM, Stephen Saine notifications@github.com wrote:

We have a penetration testing service which berates our app with traffic intended to discover some vulnerabilities. I've found that utf8-cleaner can prevent most of the exception we used to see but there is still an exception that is thrown very often.

…ctivesupport-4.0.13/lib/active_support/core_ext/uri.rb: 15:in gsub' …ctivesupport-4.0.13/lib/active_support/core_ext/uri.rb: 15:inunescape' /usr/local/lib/ruby/2.2.0/uri/common.rb: 125:in unescape' …gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/uri_string.rb: 88:invalid_uri_encoded_utf8' …gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/uri_string.rb: 23:in valid?' …gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/uri_string.rb: 15:incleaned' …gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/middleware.rb: 56:in cleaned_string' …gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/middleware.rb: 44:insanitize_env_rack_input' …gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/middleware.rb: 30:in sanitize_env' …gems/utf8-cleaner-0.2.1/lib/utf8-cleaner/middleware.rb: 21:incall'

I've found this article which talks about many things including this error:

http://yehudakatz.com/2010/05/05/ruby-1-9-encodings-a-primer-and-the-solution-for-rails/

I'm wondering if there is anything that utf8-cleaner could do to prevent this particular exception.

You guys have any further insight on the subject? Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/singlebrook/utf8-cleaner/issues/25

[sainejob]

@sbleon - I've reached out to the scanning service for the applicable request. My exception management service was unable to capture the POST body correctly. Will let you know as soon as I have the proper information for you. Thanks for your response!

[sainejob]

@sbleon Here's the curl (make sure you are operating in UTF-8 in terminal): curl 'http://localhost/myendpoint' --data-binary '[utf8=§%e2]'

Working on the test case.

[sbleon]

Thanks, @sainejob ! This is fixed in v0.2.2.

[sainejob]

@sbleon - Thanks for the fix!