Add syntax `__rule` that inlines during grammar preprocessing

[erezsh]

Suggestion

Right now inlined rules (_rule) are inlined after parsing.

It is sometimes desirable to have them inlined during grammar preprocessing, to reduce that amount of REDUCE actions (which are computationally expensive), and possibly also avoid some shift/reduce errors.

That's especially handy for templates, which often aren't designed to be rules on their own.

Describe alternatives you've considered

Starting a rule or templates with two underscores (__rule) will inline it when preprocessing the grammar.

[ThatXliner]

+1

Rules whose name begins with an underscore will be inlined into their containing rule

My intuition says that it’s going to be done when lark constructs a parser from the grammar. But for “?rule”, I think that sort of inlining should be done after parsing input.

[erezsh]

We need to distinguish between two kinds of inlining

1) At the grammar level, before building the parser

2) At the parse tree level, while parsing

[ThatXliner]

In my opinion

_rules and the like should be grammar level

?rules should be during/after parsing

[MegaIng]

@ThatXliner

No, _rules should not be at grammar level, otherwise stuff like below has terrible performance

start: _options _options

_options: a | b | c | d | e | f | g | h

Currently that's a total of 1 + 8 = 9 rules. If _rules does inlining, that's 8 * 8 = 64 rules.

[erezsh]

Also, then recursion within _rule would be impossible.

[ThatXliner]

Oh right. My bad.

I was thinking about the _templates{}. I think those should be non-recursive for most use cases.

If that may break some people’s code, why not an inline keyword?

[erezsh]

If that may break some people’s code, why not an inline keyword?

That's what the __rule / __template syntax is for.

[ThatXliner]

I just think that might break something… 🤷‍♂️