Include decorators in `Class` and `FunctionDef` range

[MichaReiser]

This PR changes the ClassDef and FunctionDef definitions to include the range of their decorators.

Example

@test
def f(): 
    pass

• Before: The functions' range was from line 2, column 0 to line 3, column 9
• New: The functions' range spans from line 1, column 0 to line 3, column 9 (includes the @test decorator)

Motivation

This change is because these are the only nodes for where the property parent.range.includes(child.range) does not hold. The fact that this property is not respected means it is e.g. impossible to use a binary search over the AST to find a node by its range.

Downsides

The main downside that I'm aware of is that diagnostics using range.start to compute the line index for functions and classes may now incorrectly point to the line of the function's decorator instead of the function definition. I don't know if this is a problem for RustPython. Ruff already uses custom infrastructure to extract the range of the function name only to limit the diagnostic range (and e.g. avoid highlighting the whole function or class including the body)

Ruff

https://github.com/charliermarsh/ruff/pull/4467

[youknowone]

Oh.. this is a Python spec omg.

Python 3.11.2 (main, Feb 16 2023, 02:55:59) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.parse("""
... @decorator
... def x(): pass
... """)
<ast.Module object at 0x100738be0>
>>> a = _
>>> f = a.body[0]
>>> f.lineno
3
>>> f.decorator_list[0].lineno
2

We will really need a feature flag for ruff.

[youknowone]

I'd better to understand it better if this is also related to symbol table or SyntaxError. If not, it can be handled in ast module with this patch

[youknowone]

@MichaReiser Do you want to make it has different value only for more-attributes feature right now? That has no blocker.

[MichaReiser]

@MichaReiser Do you want to make it has different value only for more-attributes feature right now? That has no blocker.

In my view, this stretches feature flags to the limit, or even over the limit. We would need to rename it at least. I also plan to make other range changes:

def a():
    pass

    # comment

The range of the function should include the #comment, which it does not today.

I think maintaining this change in a fork is straightforward enough and probably easier than dealing with even more conditional code that has very limited test coverage.

[youknowone]

The range of the function should include the #comment, which it does not today.

it seems going really different.

I think maintaining this change in a fork is straightforward enough and probably easier than dealing with even more conditional code that has very limited test coverage.

Yes, that sounds like truth.

[youknowone]

Reusing ast crate but making a fork of parser crate makes sense to me.

[youknowone]

We can make a new crate like rustpython-parser-util to share fundamental parser utilities except for the details.

[MichaReiser]

We can make a new crate like rustpython-parser-util to share fundamental parser utilities except for the details.

I don't think that this is necessary. I still plan to pull in upstream changes. I don't expect to many conflicts since the grammar shouldn't change frequently