no-context / moo

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
BSD 3-Clause "New" or "Revised" License
824 stars 66 forks source link

[Question] How to match a string at start and end of token without capturing groups #170

Open inferrinizzard opened 2 years ago

inferrinizzard commented 2 years ago

I am converting my custom tokenizer to moo and ran into a problem case:

// a pattern that used to look like this:
'(?<tag>\\$\\w*\\$)[\\s\\S]*?(?:\\k<tag>|$)'
// as regex literal:
/(?<tag>\$\w*\$)[\s\S]*?(?:\k<tag>|$)/

This pattern tries to match Postgres $$ strings, such as:

-- standard && string
$$lorem ipsum dolor$$
-- custom $$ string with tag (fails here)
$example$this example string fails$example$

The pattern normally works by backreferencing the captured tag between the first dollar signs to check if the second pair of dollar signs has the same tag but this breaks without the use of capturing group when lexing with moo.

Any suggestions on how to workaround this limitation ?

inferrinizzard commented 2 years ago

I see #132 , same issue here