search

amykyta3 / speedy-antlr-tool

Generate an accelerator extension that makes your Antlr parser in Python super-fast!
BSD 3-Clause "New" or "Revised" License
29 stars 7 forks source link

Provide access to hidden tokens in a stream #4

Open sergun opened 3 years ago

sergun commented 3 years ago

Hi!

First of all thanks a lot for this great tool!

My question is: I need (in Python) CommonTokenStream(lexer) class instance to get access to hidden tokens like comments with .getHiddenTokensToLeft/Right(...) methods of that stream. Am I right there is no easy way / workaround do get such instance from do_parse?

Sergey.

amykyta3 commented 3 years ago

Currently the tool only supports transcribing tokens part of the visible grammar. I purposely have it discard everything else since a lot of the processing time is actually spent translating the parsed tree & tokens into Python objects. I can look into what it would take to also transcribe these hidden tokens. I admit i have not used that feature before.

sergun commented 3 years ago

It would be nice if you can implement this.

BTW Do you have any ideas why ANTLR community still didn't implement cython backend? :-) Python backend is unusable in most cases.. And only projects like your makes it working. E.g. I deal with tens of SQL-scripts each with hundred of lines. Parsing of them in Python-backend takes minutes (((