search

datalust / superpower

A C# parser construction toolkit with high-quality error reporting
Apache License 2.0
1.05k stars 98 forks source link

Making Comments into AST (best way?) #109

Closed xDGameStudios closed 3 years ago

xDGameStudios commented 4 years ago

Referring to my previous post #108 I rephrased the problem and present here a solution! I was pretty lost dealing with this issue I wanted to keep comments in the syntax tree for latter use regarding function declaration documentation.

I ended up using a wrapper around the default Token.Equals(...) method that consumes comment tokens in an optional fashion. Then when no more comment tokens are left I try to match the token I want and now that I have the leading I can keep them in a static array and associate it with the just parsed Token.

What do you think of this approach? Do you have a better one?

matthiaszoellner commented 4 years ago

As far as I understand, the comments don't belong to a specific token, but to a specific AST node. Imagine the token stream

[DocumentationComment, Type, Identifier, OpenParenthesis, CloseParenthesis, Semicolon]

Lets assume, that this all belongs to a function declaration with returntype, functionname, argumentlist. Now the comment should also belong to the whole declaration and not specifically to the returntype of the declaration.

So I wouldn't remove the comments from the token stream, but keep them there and change the parser logic to allow a tokenstream starting with an optional comment token at appropriate points (then attach the comment to parsed node or dismiss it in places where comments are not interesting)