help: how can I use custom lexer to give multiple alternative tokens?

[vaxin]

It seems that next() just return one token, if I want to provide multiple lexical results, how can I do? For example: Apple is good. I want to use customized custom lexer to give two tokens: value: value:

I use it to tell the user the sentence has two meanings.

[vaxin]

Hi, vaxin, Maybe you don't understand the working mechanism, the lexer.next will naturally support this. If you see the code in nearley in feed(chunk) body: while (true) { try { token = lexer.next(); if (!token) { break; } }

you will see, everytime you invoke next you can give a different token, and you must maintain all the states by yourself, the tmp state will stay in two places. The first place: after lexer.reset the state can be stored through lexer.save(), and next time it will be passed to lexer.reset(). The second place: in the while loop the lexer is not reset, you can store states in the self. for stable coding, you'd better put them together. Because feed is flexible, you may see feed('a'), feed('p'), feed('p'), feed('l'), feed('l'), also you may see feed('apple') or feed('app') feed('l'). https://github.com/no-context/moo/blob/d8a76f79c310ea54143bddb486e70dcad76b1c73/moo.js#L437 the implementation of Moo is the best reference for you :)