Chevrotain / chevrotain

Parser Building Toolkit for JavaScript
https://chevrotain.io
Apache License 2.0
2.44k stars 200 forks source link

Custom Token Pattern's matched result not being consumed #1879

Closed reaperdtme closed 1 year ago

reaperdtme commented 1 year ago

Hey, I'm pretty new to this, so bear with me. What I'm doing seems relatively straightforward, and it looks like it's working as it should, but the behavior is incorrect. I have simple grammar syntax:

A tab delimited description block initiated by a "`d" followed by at least one identifier as so:

`d A mover, activator, watcher
observer {}

In this case, we're not even running into tab delineation.

My rule in pseudo code is

rule concept
  option () => consume(Description)
  at_least_one( identifier )

My custom token pattern for description correctly returns:

Exec Result [
  '`d A mover, activator, watcher\n',
  undefined,
  undefined,
  index: 0,
 ]

The parser acknowledges this but then fails with an EarlyExitException on the at_least_one(identifier) trying to match against the same description string but on line two:

EarlyExitException: Expecting: expecting at least one iteration which starts with one of these possible Token sequences::
    <[Identifier]>
  but found: '`d A mover, activator, watcher
  '

{
    token: {
      image: '`d A mover, activator, watcher\n',
      startOffset: 31,
      endOffset: 61,
      startLine: 2,
      endLine: 2,
      startColumn: 1,
      endColumn: 31,
      tokenTypeIdx: 19,
      tokenType: [Object],
      payload: 'A mover, activator, watcher'
    },
    resyncedTokens: [],
    previousToken: {
      image: '`d A mover, activator, watcher\n',
      startOffset: 0,
      endOffset: 30,
      startLine: 1,
      endLine: 1,
      startColumn: 1,
      endColumn: 31,
      tokenTypeIdx: 19,
      tokenType: [Object],
      payload: 'A mover, activator, watcher'
    },
    context: { ruleStack: [Array], ruleOccurrenceStack: [Array] }
  }
]

Offset 31 in the sample text is the 'o' in the observer, but the error message says the image is '`d' Why is the description line on line 1 being replayed as line two against my identifier match

My identifier token is a simple /[a-zA-Z]\w*/