search

antlr / antlr4

ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.
http://antlr.org
BSD 3-Clause "New" or "Revised" License
17.12k stars 3.28k forks source link

Lexers need a "self" attribute similar to $parser for Parsers. #3804

Open RossPatterson opened 2 years ago

RossPatterson commented 2 years ago

For the same reason that parsers need a self-like attribute in semantic predicates(i.e., self vs. this vs. ...), lexers do as well. Commit 38c4e2b0028d2a305d0cf05ce5669d23d67ea88e, back in 2014, added $parser for this purpose, but attempting to use it in a lexer generates error ATTRIBUTE_IN_LEXER_ACTION "attribute references not allowed in lexer actions: $parser".

The obvious choice for such an attribute is $lexer.

RossPatterson commented 2 years ago

Adding a new attribute could, of course, be a breaking change for some existing grammars.

Seaching Google for "$lexer" and "antlr" finds only hits for using a generated lexer in languages that require or prefer $whatever variables (e.g., Perl, PHP). Seaching for just "$lexer" doesn't find anything else of significance.

Grepping the grammars-v4 corpus for "$lexer" finds only _scripts/templates/PHP/Test.php, which is a case of using the generated lexer. Doing the same for the antlr4 repo finds only runtime-testsuite/resources/org/antlr/v4/test/runtime/helpers/Test.php.stg and doc/php-target.md, which are similiar cases, and runtime/Dart/lib/src/atn/src/lexer_atn_simulator.dart, runtime-testsuite/resources/org/antlr/v4/test/runtime/helpers/Test.php.stg, tool/src/org/antlr/v4/parse/ANTLRParser.g, tool/src/org/antlr/v4/parse/ATNBuilder.g, and tool/src/org/antlr/v4/parse/GrammarTreeVisitor.g, which are false positives (i.e., substring matches).

That's not an exhaustive check, but it is indicative.