search

seattlerb / ruby_parser

ruby_parser is a ruby parser written in pure ruby. It outputs s-expressions which can be manipulated and converted back to ruby via the ruby2ruby gem.
http://www.zenspider.com/projects/ruby_parser.html
476 stars 100 forks source link

Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT #302

Closed nijikon closed 4 years ago

nijikon commented 5 years ago

Per @mensfeld request, refs https://github.com/DamirSvrtan/fasterer/issues/72, I moved it here.

When running fasterer I get:

bundler: failed to load command: fasterer (/home/mencio/.rbenv/versions/2.6.5/bin/fasterer)
Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT
  gems/ruby_parser-3.14.0/lib/ruby_lexer.rex.rb:163:in `gsub'
  gems/ruby_parser-3.14.0/lib/ruby_lexer.rex.rb:163:in `block in next_token'
  gems/ruby_parser-3.14.0/lib/ruby_lexer.rex.rb:68:in `action'
  gems/ruby_parser-3.14.0/lib/ruby_lexer.rex.rb:163:in `next_token'
  gems/ruby_parser-3.14.0/lib/ruby_parser_extras.rb:1266:in `next_token'
  /home/mencio/.rbenv/versions/2.6.5/lib/ruby/2.6.0/racc/parser.rb:259:in `_racc_do_parse_c'
  /home/mencio/.rbenv/versions/2.6.5/lib/ruby/2.6.0/racc/parser.rb:259:in `do_parse'
  gems/ruby_parser-3.14.0/lib/ruby_parser_extras.rb:1301:in `block in process'
  /home/mencio/.rbenv/versions/2.6.5/lib/ruby/2.6.0/timeout.rb:93:in `block in timeout'
  /home/mencio/.rbenv/versions/2.6.5/lib/ruby/2.6.0/timeout.rb:33:in `block in catch'
  /home/mencio/.rbenv/versions/2.6.5/lib/ruby/2.6.0/timeout.rb:33:in `catch'
  /home/mencio/.rbenv/versions/2.6.5/lib/ruby/2.6.0/timeout.rb:33:in `catch'
  /home/mencio/.rbenv/versions/2.6.5/lib/ruby/2.6.0/timeout.rb:108:in `timeout'
  gems/ruby_parser-3.14.0/lib/ruby_parser_extras.rb:1289:in `process'
  gems/fasterer-0.7.1/lib/fasterer/parser.rb:8:in `parse'
  gems/fasterer-0.7.1/lib/fasterer/analyzer.rb:21:in `scan'
  gems/fasterer-0.7.1/lib/fasterer/file_traverser.rb:57:in `scan_file'
  gems/fasterer-0.7.1/lib/fasterer/file_traverser.rb:49:in `block in traverse_files'
  gems/fasterer-0.7.1/lib/fasterer/file_traverser.rb:49:in `each'
  gems/fasterer-0.7.1/lib/fasterer/file_traverser.rb:49:in `traverse_files'
  gems/fasterer-0.7.1/lib/fasterer/file_traverser.rb:26:in `traverse'
  gems/fasterer-0.7.1/lib/fasterer/cli.rb:7:in `execute'
  gems/fasterer-0.7.1/bin/fasterer:5:in `<top (required)>'
  /home/mencio/.rbenv/versions/2.6.5/bin/fasterer:23:in `load'
  /home/mencio/.rbenv/versions/2.6.5/bin/fasterer:23:in `<top (required)>'

repro:

git clone https://github.com/mensfeld/broken-fasterer.git
cd broken-fasterer
./run.sh
zenspider commented 4 years ago

Not sure what I CAN do about this... or what is reasonable when coming across a string with invalid encodings.

zenspider commented 4 years ago

Anything?

mensfeld commented 4 years ago

@zenspider if you ask me, I don't really know :man_shrugging: I would not expect a failure in utf8 vs ascii for sure but beyond that... no idea.

zenspider commented 4 years ago

This should be fixed. I'll release it soon. Thank you!

nijikon commented 4 years ago

Thanks!