Chevrotain / chevrotain

Parser Building Toolkit for JavaScript
https://chevrotain.io
Apache License 2.0
2.44k stars 200 forks source link

feat(lexer): added recoveryEnabled option to the Lexer class configuration #1839

Closed jonestristand closed 1 year ago

jonestristand commented 1 year ago

Setting recoveryEnabled to false on an instance of the Lexer class will prevent chevrotain from trying to consume more characters when it cannot match a token at the current scanning offset

fix #1838

Useful when a token does not match but may match on the next character. For instance I match 'account names' which can optionally be surrounded by parentheses, but can also contain parentheses if not fully surrounded. ex:

'Real' accounts

Assets:Chequing
Assets:Joint Chequing
(An)Asset:Chequing

'Virtual' accounts

(Assets:Chequing)
(Assets:Joint Chequing
((An)Asset:Chequing)

Due to the interspersed brackets, it isn't easy to take care of the () in the parser as separate tokens, as I'd have to concatenate the internal parentheses and text segments. In some places I want to ONLY accept a 'real' account, but when the lexer steps forward one char (losing the opening bracket), it will now match. ex:

Assets:Chequing <- valid token
(Assets:Chequing) <- invalid token in this lexing mode, BUT:
Assets:Chequing) <- will be matched when it tries to skip the opening parentheses