msujew
commented
1 year ago I'm not sure that this would be the right approach to this issue. Just to abort lexing you can probably build a lexer like this:
const AnyOtherToken = createToken({
name: "AnyOtherToken",
pattern: () => throw new Error()
})
const Digit = createToken({ name: "Digit", pattern: /[0-9]/ })
const Whitespace = createToken({
name: "Whitespace",
pattern: /\s+/,
group: Lexer.SKIPPED
})
const customPatternLexer = new Lexer([Whitespace, Digit, AnyOtherToken])This will throw an error while lexing if it encounters any character which doesn't match either the Whitespace or Digit token.
jonestristand
bd82
Useful to be able to disable recovery for the lexer. Currently it will skip input characters until it finds an offset that matches a token again, gives an error, and continues tokenizing. In some applications it would be desirable to stop lexing if no suitable token is found at the current offset.