search

seattlerb / ruby_parser

ruby_parser is a ruby parser written in pure ruby. It outputs s-expressions which can be manipulated and converted back to ruby via the ruby2ruby gem.
http://www.zenspider.com/projects/ruby_parser.html
475 stars 102 forks source link

Failing to parse string interpolation using `%Q\0` and a trailing `\n` #299

Closed spohlenz closed 4 years ago

spohlenz commented 4 years ago

I've encountered a case where ruby_parser is failing to parse string interpolation using %Q with \0 as the delimiter, when there is a trailing newline. For some background on where this arises, see k0kubun/hamlit#148.

Consider standard interpolation using %Q[]:

2.6.5 :005 > RubyParser.new.parse("%Q[\#{123}]")
 => s(:dstr, "", s(:evstr, s(:lit, 123)))

2.6.5 :006 > RubyParser.new.parse("%Q[\#{123}\n]")
 => s(:dstr, "", s(:evstr, s(:lit, 123)), s(:str, "\n"))

If we change the delimiter to \0, then the first case (no newline) works but the second results in an error:

2.6.5 :012 > RubyParser.new.parse("%Q\0\#{123}\0")
 => s(:dstr, "", s(:evstr, s(:lit, 123))) 

2.6.5 :013 > RubyParser.new.parse("%Q\0\#{123}\n\0")
Traceback (most recent call last):
       16: from /Users/sam/.rvm/gems/ruby-2.6.5/gems/ruby_parser-3.14.0/lib/ruby_parser.rb:33:in `each'
       15: from /Users/sam/.rvm/gems/ruby-2.6.5/gems/ruby_parser-3.14.0/lib/ruby_parser.rb:36:in `block in process'
       14: from /Users/sam/.rvm/gems/ruby-2.6.5/gems/ruby_parser-3.14.0/lib/ruby_parser_extras.rb:1289:in `process'
       13: from /Users/sam/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/timeout.rb:108:in `timeout'
       12: from /Users/sam/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/timeout.rb:33:in `catch'
       11: from /Users/sam/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/timeout.rb:33:in `catch'
       10: from /Users/sam/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/timeout.rb:33:in `block in catch'
        9: from /Users/sam/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/timeout.rb:93:in `block in timeout'
        8: from /Users/sam/.rvm/gems/ruby-2.6.5/gems/ruby_parser-3.14.0/lib/ruby_parser_extras.rb:1301:in `block in process'
        7: from /Users/sam/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/racc/parser.rb:259:in `do_parse'
        6: from /Users/sam/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/racc/parser.rb:259:in `_racc_do_parse_c'
        5: from /Users/sam/.rvm/gems/ruby-2.6.5/gems/ruby_parser-3.14.0/lib/ruby_parser_extras.rb:1266:in `next_token'
        4: from /Users/sam/.rvm/gems/ruby-2.6.5/gems/ruby_parser-3.14.0/lib/ruby_lexer.rex.rb:112:in `next_token'
        3: from /Users/sam/.rvm/gems/ruby-2.6.5/gems/ruby_parser-3.14.0/lib/ruby_lexer.rb:856:in `process_string'
        2: from /Users/sam/.rvm/gems/ruby-2.6.5/gems/ruby_parser-3.14.0/lib/ruby_lexer.rb:447:in `parse_string'
        1: from /Users/sam/.rvm/gems/ruby-2.6.5/gems/ruby_parser-3.14.0/lib/ruby_lexer.rb:990:in `rb_compile_error'
RubyParser::SyntaxError (unterminated string meets end of file. near line 1: "")

Using eval to run the snippet works, so it does seem to be valid usage.

2.6.5 :014 > eval("%Q\0\#{123}\n\0")
 => "123\n"
zenspider commented 4 years ago

This is both horrible and fixed.