Why aren't comments in the AST?

jgm / djot

A light markup language

https://djot.net

MIT License

1.62k stars 43 forks source link

Why aren't comments in the AST? #248

Closed alxlg closed 9 months ago

alxlg commented 9 months ago

Why comments like {% this %} are completely skipped instead of being in the AST (and ignored when converting to, for example, HTML)?

I mean, what if one wants to use a parser to develop a Djot GUI editor? How would one enable the user to see and edit comments if they aren't in the AST at all?

jgm commented 9 months ago

For an editor you'd want a concrete rather than an abstract syntax tree. Even ignoring comments, there is a many -> one mapping from concrete syntax to the AST.

matklad commented 9 months ago

To give a bit more context here, while it is possible, with some extra engineering, to re-use the same parser for CST for an IDE and AST for "real" processing, this usually results in some combination of:

AST parser being slow
CST parser not being flexible enough (esp. around error handling, or incremental reparsing)
Parsing code becoming completely unreadable due to extra abstraction

I think "have two parsers" is a reasonable starting point here.