search

pegex-parser / pegex-pm

Pegex Parser for Perl
62 stars 22 forks source link

Use double quotes for word tokens #38

Open ingydotnet opened 9 years ago

ingydotnet commented 9 years ago

ie, this:

"foo"

becomes:

/\bfoo\b/

and this:

"foo bar baz"

becomes:

/\b(?:foo|bar|baz)\b/

and maybe, we make double quotes define "tokens" instead of the special regexes. We still need to think about what tokens are and how they get used in grammars. Will create different issue for that.

Also we should probably eat leading whitespace on the regexes above. At least by default. Tokenization semantic variations can be made with new Pegex directives like: %TokenizationMethod whitespace.