whitequark / parser

A Ruby parser.
Other
1.58k stars 198 forks source link

Newlines before comments #1025

Open kddnewton opened 3 months ago

kddnewton commented 3 months ago

In the following code:

class C
  # foo
end

I would expect the newline token after the C constant to be emitted before the comment token, but it is the opposite:

irb(main):005> Parser::CurrentRuby.new.tokenize(Parser::Source::Buffer.new("", source: "class C\n# foo\nend\n"))
=> 
[s(:class,
  s(:const, nil, :C), nil, nil),
 [#<Parser::Source::Comment :2:1 "# foo">],
 [[:kCLASS, ["class", #<Parser::Source::Range  0...5>]],
  [:tCONSTANT, ["C", #<Parser::Source::Range  6...7>]],
  [:tCOMMENT, ["# foo", #<Parser::Source::Range  8...13>]],
  [:tNL, [nil, #<Parser::Source::Range  7...8>]],
  [:kEND, ["end", #<Parser::Source::Range  14...17>]],
  [:tNL, [nil, #<Parser::Source::Range  17...18>]]]]

I can work around this, but I just want to clarify what the rule is here?