incompatible character encodings: ASCII-8BIT and UTF-8 even with # encoding: utf-8

[tom-kuca]

Following examples fails with incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) even though the encoding is defined and the file is UTF-8 encoding.

# encoding: utf-8

class Article

    def run
       "ř
ř"
    end

end

The line break in the middle of string is important, everything is fine if

• there isn't the line break or
• there aren't any non-ASCII characters on either first or second line

Command:

yard -o doc /tmp/article.rb

Backtrace:

/home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard/parser/ruby/ruby_parser.rb:49:in `parse': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard/parser/ruby/ruby_parser.rb:49:in `parse'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard/parser/ruby/ruby_parser.rb:15:in `parse'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard/parser/source_parser.rb:438:in `parse'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard/parser/source_parser.rb:361:in `parse_in_order'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard/parser/source_parser.rb:89:in `block in parse'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard/logging.rb:66:in `enter_level'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard/parser/source_parser.rb:88:in `parse'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard.rb:18:in `parse'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard/cli/yardoc.rb:215:in `run'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard/cli/command.rb:13:in `run'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard/cli/command_parser.rb:68:in `run'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/lib/yard/cli/command_parser.rb:50:in `run'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/gems/yard-0.7.5/bin/yard:4:in `<top (required)>'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/bin/yard:19:in `load'
        from /home/tomas/.rvm/gems/ruby-1.9.3-p0/bin/yard:19:in `<main>'

• Ruby 1.9.3
• Yard 0.7.5

[lsegal]

Hi,

This is a bug in Ripper, so you should file a bug with http://bugs.ruby-lang.org

I can reproduce this bug directly in the ripper library (stdlib in 1.9.3) without YARD as follows:

$ ruby -v -rripper -e 'p Ripper.sexp(ARGF.read)' lib/article_with_newline.rb
ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-darwin11.0.0]
/Users/user/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/ripper/sexp.rb:31:in `parse': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
    from /Users/user/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/ripper/sexp.rb:31:in `sexp'
    from -e:1:in `<main>'

Removing the newline works:

$ ruby -v -rripper -e 'p Ripper.sexp(ARGF.read)' lib/article_without_newline.rb
ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-darwin11.0.0]
[:program, [[:class, [:const_ref, [:@const, "Article", [3, 6]]], nil, [:bodystmt, [[:def, [:@ident, "run", [5, 8]], [:params, nil, nil, nil, nil, nil], [:bodystmt, [[:string_literal, [:string_content, [:@tstring_content, "řř", [6, 8]]]]], nil, nil, nil]]], nil, nil, nil]]]]

[yb66]

Was a bug filed for this? If not (and I can't find one), I have much the same problem and will file it too.

[lsegal]

I filed a related issue http://bugs.ruby-lang.org/issues/6200 but it's for heredoc strings, not this specific issue. That said, the heredoc issue was fixed in 2.0-head when I reported, so this may very well have also been fixed. Is this still an issue in 1.9.3?

[yb66]

The problem is in ruby-1.9.2-p320. It's fixed in 1.9.3. Ok, I should've tried that first, my apologies. Thanks.

[lsegal]

I don't think 1.9.2 is maintained. If it is, you should consider opening a backport issue for 1.9.2.