Closed anesterenok closed 1 year ago
Hey @anesterenok,
Talking purely about the Langium use case, this is something that we've fixed in https://github.com/langium/langium/pull/579 and the fix is already available in one of the newer snapshot versions for the upcoming 0.5.0
release.
Basically, Chevrotain expects that any tokens that are passed to a parser instance/constructor are always part of a lexer first, so maybe we should check for that in the RecognizerEngine
code.
Hi @anesterenok
Thanks for providing a reproducible example and your analysis, this will speed up the debugging. Perhaps augmenting should done earlier in that code, except I am not sure of the side effects.
I will investigate this.
Sorry for the long delay...
Anyhow you should create Tokens using the createToken()
API, not as plain JavaScript objects.
It performs additional logic in the background and augments the Token descriptor objects with additional properties.
I can reproduce the issue with the plain token objects, however it is simply not how the API should be used So I will be closing this issue...
Running this code throws an exception:
Output:
This is happening because RecognizerEngine https://github.com/Chevrotain/chevrotain/blob/8096a8a23c0adb247913a6ad2219cf32a840d076/packages/chevrotain/src/parse/parser/traits/recognizer_engine.ts#L180
does not assume that input TokenTypes are already augmented with some technical fields. This is actually very good, because other products using Chevro (e.g. Langium) do create TokenTypes without calling createToken() or other ways.
But for multi-modal grammars, logic in https://github.com/Chevrotain/chevrotain/blob/8096a8a23c0adb247913a6ad2219cf32a840d076/packages/chevrotain/src/parse/parser/traits/recognizer_engine.ts#L143 prevents to reach line 180, because the check for isTokenType is check for tokenTypeIdx attribute existence, which is not possible before augmentation.
So basically token defs must be augmented to run augmentation in case they aren't.
I'm not so sure on what's better to be changed in that regard, so please fix as you see fit.