search

munificent / craftinginterpreters

Repository for the book "Crafting Interpreters"
http://www.craftinginterpreters.com/
Other
8.84k stars 1.04k forks source link

Lox lexer accepts texts like "54hello" #127

Closed dbarisakkurt closed 7 years ago

dbarisakkurt commented 7 years ago

I was studying your book and while I was reading code, i typed "54hello", the lexer of the language accepted it and the lexer returned:

54hello NUMBER 54 54.0 IDENTIFIER hello null EOF null

Shouldn't it reject these kind of strings since they are against our identifier rule ([a-zA-Z_][a-zA-Z_0-9]*) or number rule?

I copied your Lox, Scanner,Token and TokenType classes into my workspace. It was against my expectation and i think it should conclude in error.

iwatakeshi commented 7 years ago

I think the lexer is correctly scanning the tokens, but you're right that lox should report an error. However, I also think that it is the parser's job to report the issue and not the lexer.

Here's a repl.it example using Javascript. Running the example will cause a SyntaxError.

munificent commented 7 years ago

@iwatakeshi is right. From the lexer's perspective, the code is fine. You get two tokens, a number followed by an identifier. The lexer doesn't know that's wrong any more than it would complain on, say, )(. The parser will take care of ensuring an error is reported in this case.