effekt-lang / effekt

A language with lexical effect handlers and lightweight effect polymorphism
https://effekt-lang.org
MIT License
335 stars 24 forks source link

Improve lexer resilience #522

Open jiribenes opened 4 months ago

jiribenes commented 4 months ago

It would be really lovely to have a slightly more resilient lexer.

There's two different kinds of problems that the lexer can encounter:

  1. "completely recoverable": instead of crashing when encountering an invalid escape, just continue lexing the given string and only store the error somewhere to report it later (we don't really care about an invalid escape even in Typer!)
  2. "partially recoverable": just create a proper error token and continue lexing; the tricky part here is finding the delimiter until which the error token should be (where we can recover)
jiribenes commented 2 months ago

I think this approach isn't great: separating out soft and hard errors doesn't make things easier. Instead, I think we should:

  1. always advance
  2. produce error tokens recognised by the backend
  3. leave the actual string parsing, escaping etc. to the Parser (=> moving to a Token = { offset: Int, length: Int, kind: TokenKind } representation or something similar)

from #523