search

lark-parser / lark

Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.
MIT License
4.92k stars 417 forks source link

match_examples failing to match because trying to transform a lark-made Token #1001

Open ornariece opened 3 years ago

ornariece commented 3 years ago

match_examples, at some point

https://github.com/lark-parser/lark/blob/2335aa63e183c9182d6f8554b0d2d3714fd2286b/lark/exceptions.py#L110

creates a Token, whose value is set to an empty string:

https://github.com/lark-parser/lark/blob/2335aa63e183c9182d6f8554b0d2d3714fd2286b/lark/parsers/lalr_interactive_parser.py#L88

the problem is, this empty string might not be a valid input for whatever transformer method the user defined. in my case, i have a method that uses Decimal from decimal, and trying to pass an empty string to it results in a decimal.InvalidOperation being raised. that exception isn't caught by the except statement

https://github.com/lark-parser/lark/blob/2335aa63e183c9182d6f8554b0d2d3714fd2286b/lark/parsers/lalr_interactive_parser.py#L89

and match_examples fails because of that. with a broader except statement (say except Exception:), mach_examples succeeds

MegaIng commented 3 years ago

Well, match_examples succeeds (e.g. not fails), but does not produce the correct result.

MegaIng commented 3 years ago

A workaround that is not quite good, but at least doesn't crash is to use use_accepts=False, but I am looking into a fix.

MegaIng commented 3 years ago

Ok, this is major: When you have that kind of Transformer, the exception can't even be displayed correctly, it only shows <exception str() failed>.

MegaIng commented 3 years ago

Does this branch work: https://github.com/MegaIng/lark/tree/fix-accepts

If it works, this was surprisingly little effort.

ornariece commented 3 years ago

works for me!