Calling a token "text", "null" or "ws" in Rakudo makes matching fail

[p6rt]

Migrated from rt.perl.org#57864 (status was 'resolved')

Searchable as RT57864$

[p6rt]

From @masak

r30183​: $ ./perl6 -e 'grammar A { token TOP { \ }; token foo { d } }; "d" ~~ A​::TOP' # works $ ./perl6 -e 'grammar A { token TOP { \ }; token text { d } }; "d" ~~ A​::TOP' # fails! Null PMC access in type() [...]

Same goes for 'rule text' or 'regex test'. My uneducated guess is that something inside PGE is called 'text' and clashes.

[p6rt]

From @pmichaud

On Tue Aug 12 11​:58​:42 2008, masak wrote​:

r30183​: $ ./perl6 -e 'grammar A { token TOP { \ }; token foo { d } }; "d" ~~ A​::TOP' # works $ ./perl6 -e 'grammar A { token TOP { \ }; token text { d } }; "d" ~~ A​::TOP' # fails! Null PMC access in type() [...]

Same goes for 'rule text' or 'regex test'. My uneducated guess is that something inside PGE is called 'text' and clashes.

You're correct -- .text is a method on Match objects (see S05), and at the moment that tends to conflict with any rules named 'text' in a grammar. This may be fixed when we convert PGE to generate multimethods for regexes instead of normal methods, but ultimately there's a deep underlying conflict with Match methods and grammar methods that we'll need to resolve somehow.

Thanks,

Pm

[p6rt]

The RT System itself - Status changed from 'new' to 'open'

[p6rt]

From @chrisdolan

On Sun Sep 14 07​:51​:56 2008, pmichaud wrote​:

You're correct -- .text is a method on Match objects (see S05), and at the moment that tends to conflict with any rules named 'text' in a grammar. This may be fixed when we convert PGE to generate multimethods for regexes instead of normal methods, but ultimately there's a deep underlying conflict with Match methods and grammar methods that we'll need to resolve somehow.

It seems to be Regex.pir that is taking all of the invocations. I have discovered that, beyond \, I cannot create tokens/rules named "null", "ws", etc. The latter is particularly problematic in writing parsers in Perl6. The following simple example shows the failure​:

grammar WSOverride {   token TOP { \<tok_foo> \<.ws> \<tok_bar> };   token tok_foo { foo };   token tok_bar { bar };   token ws { [ \h | \v | '%' ]+ }; }

ok('foo bar' ~~ WSOverride​::TOP); # succeeds ok("foo\nbar" ~~ WSOverride​::TOP); # succeeds ok("foo%%%\nbar" ~~ WSOverride​::TOP); # fails

[p6rt]

From @pmichaud

On Fri, Oct 31, 2008 at 08​:47​:09PM -0700, Chris Dolan via RT wrote​:

It seems to be Regex.pir that is taking all of the invocations. I have discovered that, beyond \, I cannot create tokens/rules named "null", "ws", etc. The latter is particularly problematic in writing parsers in Perl6. The following simple example shows the failure​:

grammar WSOverride { token TOP { \<tok_foo> \<.ws> \<tok_bar> }; token tok_foo { foo }; token tok_bar { bar }; token ws { [ \h | \v | '%' ]+ }; }

ok('foo bar' ~~ WSOverride​::TOP); # succeeds ok("foo\nbar" ~~ WSOverride​::TOP); # succeeds ok("foo%%%\nbar" ~~ WSOverride​::TOP); # fails

First, note that C\< 'foo bar' ~~ WSOverride​::TOP > isn't really the correct syntax for invoking a rule -- but the synopses don't really specify what the correct syntax is yet. At the moment we do know that it's not likely to be C\< 'foo' ~~ Grammar​::rule >.

Beyond that, the problem appears to be that Rakudo is not presently creating its grammars properly -- it appears to be deriving them directly from Any instead of from a PGE grammar class of some sort. As a result, inheritance and method dispatch don't seem to be working quite properly, and we'll need to get that fixed.

Pm

[p6rt]

From @masak

Resolving this ticket, since the original problem seems to have been solved.

Beyond that, the problem appears to be that Rakudo is not presently creating its grammars properly -- it appears to be deriving them directly from Any instead of from a PGE grammar class of some sort. As a result, inheritance and method dispatch don't seem to be working quite properly, and we'll need to get that fixed.

Ticket [perl #​65474] covers this issue, along with an example of what fails.

[p6rt]

@masak - Status changed from 'open' to 'resolved'