search

amykyta3 / speedy-antlr-tool

Generate an accelerator extension that makes your Antlr parser in Python super-fast!
BSD 3-Clause "New" or "Revised" License
29 stars 7 forks source link

Separate lexer/parser grammars #5

Closed amal-khailtash closed 3 years ago

amal-khailtash commented 3 years ago

This is great tool! Quite helpful creating a faster parser with Python binding.

It seems though as this does not support separate lexer/parser grammars. When separate lexer/parser grammar files are used I see the following problem.

I have a grammar for a parser named A with the following file/grammar names:

ALexer.g4
AParser.g4

ANTLR4 Python3 target generates these python files:

ALexer.interp
ALexer.py
ALexer.tokens
AParser.interp
AParser.py
AParser.tokens

ANTLR4 Cpp target generates these cpp files:

ALexer.cpp
ALexer.h
ALexer.interp
ALexer.tokens
AParserBaseVisitor.cpp
AParserBaseVisitor.h
AParser.cpp
AParser.h
AParser.interp
AParser.tokens
AParserVisitor.cpp
AParserVisitor.h

Running speedy_antlr_tool.generate and passing AParser.py as argument generates:

sa_a.py

sa_A_cpp_parser.cpp
sa_A_translator.cpp
sa_A_translator.h

speedy_antlr.cpp
speedy_antlr.h
speedy_antlr.o

It seems the templates require a visitor header file with the name ABaseVisitor.h, but the CPP source includes a header named: AParserBaseVisitor.cpp:.

In file included from sa_a_cpp_parser.cpp:16:
sa_a_translator.h:8:10: fatal error: ABaseVisitor.h: No such file or directory
 #include "ABaseVisitor.h"
          ^~~~~~~~~~~~~~~~
amal-khailtash commented 3 years ago

It seems this had been mentioned before in Issue #2, but I don't see a resolution, but somehow the error went away. Maybe some manual renaming of generated filenames were done?

amykyta3 commented 3 years ago

Good point. This project was motivated by the systemrdl-compiler project, where I combined the lexer & parser in a single grammar file. It shouldn't be too difficult to extend support for this to support grammars that are split across files.

amal-khailtash commented 3 years ago

On this same note. It would be great if you could support lexer only grammars as well. I have one that I just use ANTLR4 lexer only grammar.

amykyta3 commented 3 years ago

Just published v1.1.0. This adds support for grammars that are split into separate lexer/parser files. See example: https://github.com/amykyta3/speedy-antlr-example/blob/master/src/spam/parser/generate_parsers_split.sh

Regarding the lexer-only request, I have opened a separate issue here: #6

amal-khailtash commented 3 years ago

This is great. I confirm this fixes the separate grammars. Thanks.