Closed dmelcer9 closed 1 year ago
>>> from python_parser import python_parser3
>>> code = r"""for _ in foo:
... pass
... """
>>> import rich
>>> rich.print(python_parser3.parse(code))
file_input
└── for_stmt
├── var
│ └── name
│ └── _
├── var
│ └── name
│ └── foo
├── suite
│ └── pass_stmt
└── None
Very weird, going to look at the code when I get in and figure out a more complete example where it goes wrong.
Okay. I'm closing the issue for now, but feel free to re-open if you think it's worthwhile.
The issue appears to be in how the earley parser specifically interacts with the grammar:
from lark import Lark
from lark.indenter import PythonIndenter
l = Lark.open_from_package("lark", "python.lark", ['grammars'], parser="earley", postlex=PythonIndenter(), start="file_input")
print(l.parse("for _ in foo:\n pass\n"))
Results in:
lark.exceptions.UnexpectedToken: Unexpected token Token('UNDERSCORE', '_') at line 1, column 5.
Expected one of:
* TRUE
* BIN_NUMBER
* TILDE
* __ANON_24
* NONE
* STRING
* MATCH
* NAME
* FALSE
* OCT_NUMBER
* PLUS
* LPAR
* FLOAT_NUMBER
* CASE
* DEC_NUMBER
* MINUS
* LBRACE
* LSQB
* STAR
* IMAG_NUMBER
* HEX_NUMBER
* LONG_STRING
* AWAIT
@erezsh I am unable to re-open the PR from github's interface for some reason
The problem is lexer="basic"
, which gets chosen automatically when you use postlex and Earley. The python grammar is designed for parser=lalr, lexer=contextual.
@MegaIng I think you're right.
However, @dmelcer9 , your solution won't fix it, since there are other uses of NAME throughout the grammar, which means that code with "as _" still won't work, etc.
Possibly the easiest solution would be to remove the line
| "_" -> any_pattern
And then handle this case after the parse is done.
But I'm not sure it's worth it. Parsing Python with Earley isn't really a common use-case. It's only in the examples to show that it's possible.
Previously, the example Python grammar wouldn't parse text like: