wyvernlang / wyvern

The Wyvern programming language.
http://wyvernlang.github.io/
GNU General Public License v2.0
556 stars 65 forks source link

Fix Lexer flagTok not always reset #362

Closed zyx-billy closed 4 years ago

zyx-billy commented 4 years ago

Currently, the lexer's flagTok is not always set for each new (non-empty) line. This causes dslIndents to be sometimes wrongly lexed as regular indents. The following example produces an INCONSISTENT_INDENT error on the last line when it should be legal.

type Option = option.Option
if (true)
    stdout.print("hello\n")

The error is caused by the second line not resetting flagTok from TYPE to null. In this case we cannot manually set flagTok to null in any production rule because we risk overwriting the flagTok set by the builtin keywords of this line (e.g. def, type). The proposed fix adds a boolean member flagTokSet to track whether we manually set flagTok to a non-null value on this line. If not, we should revert it to null when done with this line so that the flagTok from the previous line does not get propagated any further.

(Technically we can also remove any code that manually sets flagTok to null in the lexer since it'll be covered by this final flagTokSet for each line).

JonathanAldrich commented 4 years ago

Looks great, thanks for the fix!