Add comment in AST

[black-desk]

To use djot.js as a djot formatter, we need to add comments node in AST to keep comments in the output.

[jgm]

Well, it's an abstract syntax tree, not a concrete syntax tree, and these usually don't contain comments.

[jgm]

@matklad any thoughts on this issue? See also https://github.com/jgm/djot/issues/67 We will probably revamp comments in some way.

[matklad]

This is a design question:

• Either djot syntax tree is full-fidelity, such that print . parse = id
• Or djot sytnax tree only preserves semantics: different source code yields the same tree

My gut feeling is that, long term, it is better if you have separate trees and parser -- semantics-preserving AST parser for actual intended use-case (rendering djot documents as html/pdf/etc) and a syntax-preserving parser for tooling. You can do both, but that increases complexity of the implementation, and also increases complexity for the user.

Eg., if we add comments to djot.js AST, then filters would have to care about commetns.

IIRC, someone is buildign a tree sitter grammer for djot? I'd probably try to build a formatter on top of that.

[jgm]

For the treesitter grammar see https://github.com/jgm/djot/discussions/280

[jgm]

To have full-fidelity, we'd need to represent a huge number of things beside comments. E.g., exactly how many spaces are there before and after the bullet, which bullet did you use, etc. So this would be a massive change, not a minor one. But perhaps one could argue that comments are an intermediate case and motivate including comments in the AST without pressing for full fidelity.