no-context / moo

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
BSD 3-Clause "New" or "Revised" License
814 stars 65 forks source link

Fix saving and resetting queued groups #176

Open ictrobot opened 2 years ago

ictrobot commented 2 years ago

Currently Lexer.prototype.reset and Lexer.prototype.save reference three queued fields: queuedToken, queuedText (added in #169) and queuedThrow.

There only appear to be references to queuedText and queuedGroup elsewhere, so this MR removes queuedToken and queuedThrow and adds queuedGroup to reset and save.

The below example shows how this could be problematic - on master it prints a token when it probably shouldn't:

const moo = require("./moo")

const lexer = moo.compile({
    number: {match: /\d+/},
    text: moo.fallback
})

lexer.reset('test123')
lexer.next()
lexer.reset()
console.log(lexer.next())