no-context / moo

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
BSD 3-Clause "New" or "Revised" License
821 stars 65 forks source link

Expose matched `text` #68

Closed tjvr closed 7 years ago

tjvr commented 7 years ago

There's currently a subtle edge-case when using Moo and nearley together. As a convenience, nearley allows matching tokens based on their value:

You can still use raw strings, but they will only match full tokens parsed by Moo. This is convenient for matching keywords.

ifStatement -> "if" condition "then" block

The edge case is that nearley only checks the value, not the type. So it's very easily to accidentally match, e.g., a quoted string with the value if!

This is only a problem when using capture groups (now value transformers). #63

Solution: expose text (the contents of the entire untransformed match), and modify nearley to use that instead.

tjvr commented 7 years ago

Closed via abba951.