Doc comment syntax for copying a passage of documentation from elsewhere

[srawlins]

Dartdoc currently has a system that uses two doc comment directives, @template and @macro. A block of doc comment text can be wrapped in {@template a.b.c}...{@endtemplate} in order to store it away as an insertable block of text keyed with "a.b.c". The text can be inserted elsewhere then via {@macro a.b.c}. This system has a major drawback: in order to perform a single insertion, e.g. inserting the text for "a.b.c", dartdoc must parse the doc comments of every member in the packages that may be documented. This is hundreds of thousands of elements in the case of Flutter. This prevents any kind of incremental doc generation, hurts discoverability, makes typos easy. Additionally, there is no substitution performed, so the text is not really a "template", and it's insertion is not really a "macro" application. It's just a block of static text that can be inserted here and there.

OK, so I'd like to design a new system from the ground up, supported in analyzer, and in dartdoc via analyzer API.

Requirements

• A block of doc comment text can be identified as a "passage," making it pasteable into other doc comments.
• A passage is identifiable with an unambiguous identifier. This identifier probably contains the passage's containing library's URI, in the form of a package: URI.
• A single doc comment can contain multiple passages. Again, each is identifiable with an unambiguous identifier.
• Passages can contain comment references, which are resolved at the scope of the declaration of the passage.
• Other doc comment directives, like {@youtube} are resolved at the declaration of the passage.
• No variable replacement support is necessary.
• No support for a passage within a passage is necessary.

(I'm not married to "passage," but I do like it. I'm really not a fan of "template". Other choices include snippet, chunk, excerpt, ...)

API mockup

With these requirements, we have some options of how to refer to a passage via a fully qualified name. But I imagine it contains the library's URI (package:flutter/widgets.dart) (note to take the unquoted forms into account), the passage's immediate "name" (like "foo"), and possibly the name(s) of the library elements, like class name / method name. Does a passage's name have to be unique across a library, or only within the doc comment in which it is declared? What delimiters to use between the library URI and other names?

To go lightweight, we can say that a passage's name has to be unique within it's library, and we can use a hashmark as a delimiter, so that a passage named "foo" in the Flutter widgets library is referable via "package:flutter/widgets.dart#foo". I like the hashmark as it is reminiscent of an HTML fragment delimiter. (With the new unquoted syntax, it would look something like flutter/widgets#foo.

As one possible option for how to define a passage, and how to insert a passage:

• Define a passage via {@passage name}...{@end-passage}. The name cannot contain whitespace (to be consistent with other doc directive restrictions), although maybe within quotes??? Maybe not.
• Insert a passage via {@insert URI#name}, {@excerpt URI#name}, @{include URI#name}, something like that...

Features

These requirements can enable the following features:

• In the IDE, a passage insertion is ctrl+clickable to go to the passage's definition.
• In the IDE, an unknown passage insertion can be called out with a static warning.
• In the IDE, doc comments, wherever shown, can include inserted passages.
• Dartdoc can perform replacements.

[srawlins]

CC @gspencergoog @goderbauer

[gspencergoog]

The name cannot contain whitespace (to be consistent with other doc directive restrictions), although maybe within quotes??? Maybe not.

I'd say not. Programmers are pretty familiar with snake case or camel case names, and allowing spaces adds quoting, which adds escaping and unescaping, which makes it all a lot more complex.

[gspencergoog]

(I'm not married to "passage," but I do like it. I'm really not a fan of "template". Other choices include snippet, chunk, excerpt, ...)

It's a reasonable name. I think I'd prefer excerpt, but passage works. I'd say no to snippet, because of confusion with code snippets.

[lrhn]

Would it be enough to have a qualified template reference? So instead of {@macro foo.bar} you can write {@macro ../file.dart#foo.bar}, treating everything up to the next } as a relative URI if it has a fragment, and as a normal name of not (no whitespace in either case). If you use the qualified reference, then the lookup only has to check one file, and won't find templates anywhere else. It still has to check the entire file, but I think it would have to fit snippets too.

Maybe allow templates to be named {@template #foo.bar} in which case it only works with qualified references. Inside the same file, it would be just {@macro #foo.bar}

[srawlins]

I like those ideas! I'd like to look up some stats on Flutter's use of {@macro}, like

• how many reference a template from the same library?
• how many times does a doc comment contain more than one template?

[bwilkerson]

I have a slightly different suggestion for the reference. Given a passage/excerpt named 'name', write the reference as {@insert name}, but require that there is a docImport for the library that defines the passage to be inserted.

On a different note we have just removed most of DeclarationTracker (the class that was co-opted to support 'template's in the analyzer). This arc of work would allow us to remove the rest of that class, which would simplify the code and might improve performance.