I wonder if we can remove the lookahead from the lexer entirely and move lookahead logic purely into the parser in the places where it needs it?
As I understand it, lookahead is only required for a few things (arrow functions and TS type params mostly) but currently every single time we get a token (any token) we have to pass through this code. I don't know the parser well enough to know how difficult it'd be to move the logic though.
@lucab mentioned in https://github.com/oxc-project/oxc/pull/4534 that checking lookahead came up as a hotspot when he was profiling the lexer.
I wonder if we can remove the lookahead from the lexer entirely and move lookahead logic purely into the parser in the places where it needs it?
As I understand it, lookahead is only required for a few things (arrow functions and TS type params mostly) but currently every single time we get a token (any token) we have to pass through this code. I don't know the parser well enough to know how difficult it'd be to move the logic though.