Lexer rewrite - Githubissues

KubaP / glsl-lsp

A language server implementation for the OpenGL Shading Langauge, written in Rust.

12 stars 1 forks source link

Lexer rewrite #1

Closed KubaP closed 2 years ago

KubaP commented 2 years ago

Replace the current chumsky lexer with a handwritten lexer.

This is done for the following reasons:

The existing lexer, especially the number-parsing logic was getting extremely long and with many many branches. The rewrite has much less branching and supports more valid notations.
The preprocessor parsing was only half functional, and it didn't deal with newline escapes. This was something that'd be difficult to implement. I was running into numerous cases where chumsky didn't like the ambiguity of the patterns, and the rewrite fixes this.