Instagram / LibCST

A concrete syntax tree parser and serializer library for Python that preserves many aspects of Python's abstract syntax tree
https://libcst.readthedocs.io/
Other
1.54k stars 190 forks source link

Unhelpful error message in `ParserSyntaxError` #1028

Open Zac-HD opened 1 year ago

Zac-HD commented 1 year ago

Passing this on from https://github.com/HypothesisWorks/hypothesis/issues/3759:

# repro.py
import libcst

source = "def f(*):\n pass\n"
libcst.parse_module(source)

compile(source, "", "exec")  # for comparison
$ python repro.py 
Traceback (most recent call last):
    libcst.parse_module("def f(*):\n pass\n")
  ...
libcst._exceptions.ParserSyntaxError: Syntax Error @ 2:5.
parser error: error at 1:8: expected one of ,, NAME

 pass
     ^

# and if we comment out the `parse_module()` call, we see Python gives us:
Traceback (most recent call last):
    compile(source, "", "exec")
  ...

    def f(*):
          ^
SyntaxError: named arguments must follow bare *

The error message we get from libcst points to 1:8, i.e. the close-paren of the function definition, but the exception metadata instead points to the end of the function body (for def f(*): ... it points to the start of the def). Additionally, we see expected one of ,, NAME, which suggests that the message is constructed from three alternatives of which two have an empty string representation.

I've included the error raised by Python 3.10 for comparison; it points directly to the * and has a presumably-hand-written message explaining what's going on. Unclear how much of that is worth adopting or copying over, but testing that the location of syntax errors is compatible might be worthwhile?

zsol commented 1 year ago

I completely agree the parser error messages are often downright misleading as opposed to helpful. It would be great to improve them, but currently the parser generator library we use make it difficult to emit high-quality error messages. See https://github.com/kevinmehall/rust-peg/discussions/276 and https://github.com/kevinmehall/rust-peg/pull/289 for improving that situation.