ReadToken treats comments as line continuators

[rprichard]

I suppose it's harmless, but this seems unexpected?

myvar # Continue on the next line.
      # Keep going...
= foo

rule echo
  # We want to continue ignoring this line, though.
  # It's tested in ParserTest.IgnoreIndentedComments.
  command = echo $out

build $myvar : echo

Output:

$ ninja 
[1/1] echo foo
foo

A line-continuation comment isn't accepted in places where a path or binding value are parsed. AFAICT, a binding's = symbol is the only place where a comment can be used as a line-continuation.

[rprichard]

This issue also manifests when Ninja is trying to read a newline token. e.g. Ninja rejects this:

rule echo # end-of-line comment
  command = echo $out
build foo : echo

output:

$ ninja
ninja: error: build.ninja:2: expected newline, got indent

However, it accepts this:

rule echo # end-of-line comment

  command = echo $out
build foo : echo

output:

$ ninja
[1/1] echo foo
foo

[rprichard]

Ninja rejects this:

rule r
  command = r

  generator = 1

It's tested in manifest_parser_test.cc -- grep for input:4: unexpected indent.

Maybe this issue can be fixed by:

• Stop consuming the newline in ReadToken by using a lookahead assertion, e.g.:

  comment = "#"[^\000\n]*;
  [ ]*comment / "\n" { continue; }

• Add a special Lexer::PeekIndent() method with ReadToken's current comment-line-skipping behavior.

PeekIndent duplicates some of ReadToken, which is a bit unfortunate. It seems to work, though.