search

lark-parser / lark

Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.
MIT License
4.86k stars 413 forks source link

Register user functions to avoid rule name collisions #1066

Open plannigan opened 2 years ago

plannigan commented 2 years ago

Suggestion Currently, getattr() is used on the Transformer or Visitor to find a user defined method with the same name as the Tree.data or Token.type. However, this can lead to collisions with lark methods (ex. "transform") or python keywords (ex. "from"). Adding a register_user_function() method would allow users to work around these collisions without requiring them to change the name of their grammar rules.

Additional context I've already run into this issue with my own use case and having a rule named "from" made sense. So I already have a scrappy solution that could be fleshed out and integrated into the library.

erezsh commented 2 years ago

The usual recommendation in Python regarding keywords like "from" is to write it as "from_". I think it's a reasonable solution to these collisions.

I'm not saying no, but your suggestion feels a little awkward. I'm more inclined to consider a decorator, like:

class MyTransformer(Transformer):
    @call_for("from")
    def from_(self, args): ...
plannigan commented 2 years ago

I agree that a decorator would be a cleaner interface. However, I'm not clear on how that would get implemented. I tried looking at the implementation of @v_args as an example, but there are enough levels of indirection that I wasn't able to follow how it all pieced together.

erezsh commented 2 years ago

Yeah, it's a little tricky. But it's easier than it might seem.

We handle reading the transformer in this method: https://github.com/lark-parser/lark/blob/master/lark/parse_tree_builder.py#L352

We can sweep its members at the beginning, and rename whatever is flagged. (or build a dict to map them)

plannigan commented 2 years ago

Ok, that helped. I was able to experiment and get a POC working for a decorator (I also learned about __get__()). Would you be interested if I fleshed out the POC into a PR?

erezsh commented 2 years ago

Yeah, open a PR.

Btw I expect to have a bit more free time soon, to spend on merging existing PRs.