Chevrotain / chevrotain

Parser Building Toolkit for JavaScript
https://chevrotain.io
Apache License 2.0
2.44k stars 199 forks source link

Cannot recovery by insert token just before EOF #2006

Closed hackwaly closed 5 months ago

hackwaly commented 7 months ago

Because EOF never appears in follows

  canRecoverWithSingleTokenInsertion(
    this: MixedInParser,
    expectedTokType: TokenType,
    follows: TokenType[],
  ): boolean {
    if (!this.canTokenTypeBeInsertedInRecovery(expectedTokType)) {
      return false;
    }

    // must know the possible following tokens to perform single token insertion
    if (isEmpty(follows)) {
      return false;
    }

    const mismatchedTok = this.LA(1);
    const isMisMatchedTokInFollows =
      find(follows, (possibleFollowsTokType: TokenType) => {
        return this.tokenMatcher(mismatchedTok, possibleFollowsTokType);
      }) !== undefined;

    return isMisMatchedTokInFollows;
  }
bd82 commented 5 months ago

Hmm, I don't think EOF can be automatically added to all follows set. Not all parsing rules may terminate with EOF, normally it is only the root one.

Can you try explicitly adding $.CONSUME(EOF) at the end of your root rule? Does it solve your issue?

    $.RULE("json", () => {
      $.OR([
        { ALT: () => $.SUBRULE($.object) },
        { ALT: () => $.SUBRULE($.array) },
      ]);

      // explicitly consume End Of File.
      $.CONSUME(EOF);
    });
bd82 commented 5 months ago

closing, re-open if this workaround does not work.