lark-parser / lark

Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.
MIT License
4.64k stars 397 forks source link

Checking for allowed tokens with accepts() triggers transformer callbacks #1369

Closed Daniel63656 closed 8 months ago

Daniel63656 commented 8 months ago

I use a transformer to get callblacks whenever a rule finishes. I use an interactive parser. When I check for the next tokens that can be parsed, the callbacks are triggered several times:

from lark import Lark, Token, Transformer

grammar ="""
    sentence: noun verb noun
            | noun verb "like" noun

    noun: adj? NOUN
    verb: VERB
    adj: ADJ

    NOUN: "flies" | "bananas" | "fruit"
    VERB: "like" | "flies"
    ADJ: "yummy"
"""

class MyTransformer(Transformer):
    def adj(self, children):
        print("Callback for adj:", children)
        return children

    def noun(self, children):
        print("Callback for noun:", children)
        return children

    def verb(self, children):
        print("Callback for verb:", children)
        return children

parser = Lark(grammar, parser='lalr', start='sentence', transformer=MyTransformer())
interactive = parser.parse_interactive()

#parser.parse('yummy')
#interactive.accepts()

interactive.feed_token(Token('ADJ', 'yummy'))
interactive.feed_token(Token('NOUN', 'fruit'))
print("----FROM HERE ON NO CALLBACKS SHOULD HAPPEN----")
interactive.accepts()

Is this a bug?

erezsh commented 8 months ago

Works fine for me.

Are you using the latest Lark version? i.e. 1.1.8

Daniel63656 commented 8 months ago

my version is lark-parser 0.12.0 pypi_0 pypi

which seems to be the newest version. If I run $ pip install --upgrade lark-parser I get: Requirement already satisfied: lark-parser in c:\users\daniel\anaconda3\envs\musicml\lib\site-packages (0.12.0)

Code output is:

Callback for adj: [Token('ADJ', 'yummy')] ----FROM HERE ON NO CALLBACKS SHOULD HAPPEN---- Callback for noun: [[Token('ADJ', 'yummy')], Token('NOUN', 'fruit')] Callback for noun: [[Token('ADJ', 'yummy')], Token('NOUN', 'fruit')] {'VERB'}

erezsh commented 8 months ago
pip uninstall lark-parser
pip install lark -U
Daniel63656 commented 8 months ago

Thanks a lot that fixed it. Maybe consider removing the wheels for lark-parser to avoid confusion?