Conflicts in optional production with leading optional

[ericprud]

Cc: @RubenVerborgh

This grammar yields conflicts on p2_Opt and Flag_Star:

Foo:
       p1 p2_Opt EOF    { return { p1: $1, p2: $2 } } ;

p1:
      'P1' ;

p2_Opt:

    | p2 ;

p2:
      Flag_Star 'P2'    -> $1.concat(' ', $2)
    ;

Flag_Star:
        -> '0th flag'
    | Flag_Star 'Flag'  -> $1.concat(' ', $2)
    ;

If I change Foo to be

Foo:
      p1 EOF    { return { p1: $1, p2: 'default p2' } }
    | p1 p2 EOF { return { p1: $1, p2: $2 } } ;

the error messages go away and the parser works, but that means my parser requires a lot of idiosyncratic hand maintenance. I suspect that the calculation of the state is examining only the current production when it should recursively examine the referenced productions. Note that this grammar is valid in yacc:

Foo:
      p1 p2_Opt ;

p2_Opt:

    | p2    ;

p1:
    TOK_P1  ;

p2:
    Flag_Star TOK_P2    ;

Flag_Star:

    | Flag_Star TOK_Flag    ;

with the obvious lex:

([\t\n\r ])+
"P1"        { return TOK_P1; }
"P2"        { return TOK_P2; }
"Flag"      { return TOK_Flag; }
<<EOF>>     { yyterminate();}