search

alecthomas / participle

A parser library for Go
MIT License
3.45k stars 186 forks source link

How can I use the matchLongest lexer option? #406

Closed wrycode closed 4 months ago

wrycode commented 5 months ago

It's a private field and I don't see a way to set it

alecthomas commented 4 months ago

It looks like dead code, I'll remove it. What are you trying to achieve?

wrycode commented 4 months ago

My project needs to parse input with user-defined 'form' dictionaries where a form could be one or multiple characters from the input stream, and the largest should be selected first. It's for phonetics. One example is 'kw', which makes the 'q' form, where 'k' and 'w' are also individual forms when they're not found together.

Struct tags can't be changed at runtime, so I was just converting each 'form' to a token in the lexer. It's possible to generate regex like kw|(k?w) but it gets kind of messy. Now my plan is to generate a prefix tree data structure from the user's form dictionary and just write my own parser that traverses the prefix tree while scanning the input to find the longest match.

I appreciate your work on this library.