"Void Terminal" declarations

[kjosib]

Main Idea

In any given language, there are likely to be terminals which contribute only syntactic, not semantic information. Examples include most punctuation and reserved words. Let's call these "void terminals", because the type of their semantic content is -- well -- vacuous.

If we have a list of (or criterion for recognizing) void terminals, then there's no need to explicitly filter them out of the set of right-hand-side symbols relevant to a parse-action. This could result in cleaner-looking production rules.

For Example:

• Before:
  stmt -> IF .expr THEN .stmt ELSE .stmt :test
• After:
  %void IF THEN ELSE
  in a declarations block, and later
  stmt -> IF expr THEN stmt ELSE stmt :test

Expanding on the Idea:

• Quoted-terminals like '[' or '==' are generally void, but this may depend on the application. It may be worth allowing something like %void %quoted as a declaration to indicate this.
• A list of reserved words, local to the grammar definition, would be a convenient enhancement. This would free the application from having to embed that list, and also make it evident to a human reader of the definition document. Details would be the subject of another issue.
• A related declaration like %token could allow you to opt into strictness: if it appears, then every grammar symbol must either be declared somehow as a terminal or be a non-terminal with production rules reachable from the start symbol(s).
• It may be possible for the parser-generator to exploit the fact that certain symbols never have a semantic value to decrease the size of, or activity on, the parse-stack, but this is unlikely to be a practical benefit.

Relatedly:

This suggests perhaps "Precedence" is the wrong name for its block. Perhaps it should be called "Declarations", of which precedence and associativity rules are just one kind.

Caveat: This is not, and cannot be, a perfect be-all, end-all solution. But it should make a big dent in the vast majority of

[kjosib]

This concept is now supported on the miniparse side with the void_symbols parser attribute. While I see how to add it to the macroparse side, it seems the best approach might be to reconsider the design of the macroparse module in light of certain recent developments.

[kjosib]

The "void token" idea does have an interaction with the macro system: if a macro's production rule fails to explicitly specify semantic capture positions, then textual substitution of different tokens could yield inconsistent behavior. The system ought to decide which positions in macro-rules to capture once only, rather than again for each instantiation of the macro. And indeed this is how it works.