Open jiribenes opened 4 months ago
I think this approach isn't great: separating out soft and hard errors doesn't make things easier. Instead, I think we should:
- always advance
- produce error tokens recognised by the backend
- leave the actual string parsing, escaping etc. to the Parser (=> moving to a
Token = { offset: Int, length: Int, kind: TokenKind }
representation or something similar)
from #523
It would be really lovely to have a slightly more resilient lexer.
There's two different kinds of problems that the lexer can encounter: