Closed ZedThree closed 1 year ago
I think I have a fix for all of these except the last problem, by explicitly consuming the newline at the end of a comment:
comment: $ => token(seq('!', /.*/, '\n')),
and then handling _end_of_statement
in the external scanner:
if (valid_symbols[_END_OF_STATEMENT]) {
if (lexer->lookahead == ';'
|| (lexer->lookahead == '\n' && !continued_line)
|| lexer->eof(lexer)) {
skip(lexer);
lexer->result_symbol = _END_OF_STATEMENT;
return true;
}
if (lexer->lookahead == '!' && !continued_line) {
lexer->result_symbol = _END_OF_STATEMENT;
return true;
}
}
Here we also treat comments as ending statements, unless we're "inside" a line continuation. I'll tidy up the code a bit and make a separate PR for it
@ZedThree is this issue resolved now? All three examples parse for me.
Yep, sorry I didn't update the PR comment to automatically close this
Aside from preprocessor directives, the main issue that seems to be left with the corpus I've been looking at is line continuations and end-of-statements. There are a few different issues, but they're all related and I suspect need some clever handling in the external scanner. Just to be clear, all these issues involve parsing errors, i.e. they all contain an
(ERROR)
node.The first one seems pretty innocuous, but I've not managed to solve it. It's just a
write
that only uses aformat
via a label and that label appears on the next line:Any statement or comment on a line in-between the
write
andformat
fixes the issue. It looks like the new line is getting eaten before being parsed as_end_of_statement
and then1 format()
can't be parsed as anoutput_item_list
. This can be confirmed by putting a comma after the1
, and then this is successfully (but incorrectly!) parsed as a completeoutput_item_list
.The second one is far more common (probably accounts for most of the ~200 files that error in WRF):
Here the issue is the next line starts with a comma. If the comma is moved to the preceding line or the comment removed, then it works fine.
Possibly the same as previous issue:
Again, removing the comment fixes things.
The last one involves concatenating a string literal with a line continuation in it:
This seems to only trigger an error if there's a line continuation before the first character in the literal.
My current thinking is to have a
bool continued_line
in the scanner that tracks whether we're still "in" a line continuation or not.