add full-parser, like full-lexer

benmkw commented 1 year ago

Currently it does not seem to be possible to get an AST with comments attached?

youknowone commented 1 year ago

That will be a good feature if you need one. Since I don't think you will need NonLogicalNewline, maybe just comment feature?

benmkw commented 1 year ago

I tried looking into ways to add comments but could not quickly figure out how to best add it to the grammar.

Then I looked at ruff cause they do formatting now as well and use this parser (although it seems they vendor/ fork it) and here is some code which does it for them https://github.com/astral-sh/ruff/tree/9cb5ce750e9dead53eb8235e8cfb7cff653a1cd2/crates/ruff_python_formatter/src/comments ( https://github.com/astral-sh/ruff/blob/9cb5ce750e9dead53eb8235e8cfb7cff653a1cd2/crates/ruff_python_formatter/src/comments/mod.rs#L319-L332 ) but not in a way which integrates into the parser but it seems to be retrofitted after parsing as far as I can see (maybe adding it the grammar is not trivial after all?).

Since I don't think you will need NonLogicalNewline, maybe just comment feature?

Yes that would be enough for me, it does not have to be a perfect reproduction, but I'd want to preserve the meaning/ content of the comments.

MichaReiser commented 1 year ago

Fitting comments into the grammar will be difficult because comments can appear in almost all positions.

Ruff's comment handling is specific to the formatter and not a general purpose solution.

I recommend taking a look at how oxc handles comments

An entirely different approach is to use a cst similar to rome/biome/Roslyn/swift

RustPython / Parser