Steps to remove documentation post-processing scripts

[ben-albrecht]

We have 2 post-processing scripts that get called after chpldocumenting the modules:

modules/dists/fixDistDocs.perl: Removes all class documentation except for the top distribution and removes all method & field documentation.

modules/internal/fixInternalDocs.sh: Removes or replaces specific patterns in documentation, e.g. removing symbols with chpl_ or _ prefixes.

Both of these scripts were intended to go away as newer chpldoc features make them obsolete. This issue is an attempt to capture what those chpldoc features are and what they might look like, as well as some potential short term solutions to removing these scripts until those feature are added.

Dist Docs

In the short term, the fixDistDocs.perl changes could be accomplished by adding pragma "no doc" to all of the classes, except the top distribution, and adding pragma "no doc" to each method & field in the top distribution.

In the long term, chpldoc could accomplish this more elegantly with an option to document only the top doc-string of a module, record, or class, e.g. pragma "top doc", pragma "no-recurse doc" or something else. Specific fields or methods could be explicitly documented within a pragma "top doc" via the reverse of pragma "no doc": pragma "doc", e.g.

// BlockDist.chpl

pragma "top doc"
/* 
  Only this comment is documented. 
  No fields or methods will be documented unless explicitly annotated with `pragma "doc"`
*/
class Block { ... }

pragma "no doc"
/* Not documented at all */
class LocBlock { ... }

pragma "no doc"
/* Not documented at all */
class BlockDom { .. }

...

Internal Docs

All of the fixInternalDocs.sh changes boil down to a call to awk or sed.

This could be supported in chpldoc by adding a string replacement argument or config file for string replacements (supporting regex).

Alternatively, the symbol-renaming could be accomplished in source code via annotations. I don't have any good ideas on how that might look...

The "top doc"-string option mentioned earlier could also be helpful for omitting symbols by default, since internal modules contain mostly internal symbols, e.g.

pragma "top doc"
/* 
  This will be documented, but nothing inside it will be documented, 
  unless explicitly annotated 
*/
module Internal {

  /* This will not be documented */
  proc chpl_foo()  { return 1; }

  pragma "doc"
  /* This will be documented, because it was explicitly annotated */
  proc foo() { return chpl_foo(); }

}

[lydia-duncan]

In the case of the chpl_ or _ prefixed functions, I think it will be equally easy to explicitly ignore them within chpldoc itself like we do for internally generated functions.

I'm a bit against adding something like a pragma "doc" or pragma "top doc". To me, the desire for solutions like that in user code would indicate that the symbols in question should be grouped together into an un-documented helper module or declared as private where appropriate. I think we only want it because of our internal modules are unsurprisingly full of internal details.

[ben-albrecht]

In the case of the chpl or prefixed functions, I think it will be equally easy to explicitly ignore them within chpldoc itself like we do for internally generated functions.

I like this approach too, but I don't see how it can handle the cases like layouts and distributions.

Also, note that there are several other patterns beyond _ and chpl_ that are removed or replaced in fixInternalDocs.sh.

To me, the desire for solutions like that in user code would indicate that the symbols in question should be grouped together into an un-documented helper module or declared as private where appropriate.

Fair point. Is this possible for us to do with the internal modules as well? I'd prefer this over new pragmas if it is feasible.

[lydia-duncan]

I think it is certainly possible. Definitely not for this release, and we would probably want to put this before the general group before acting on it (to hopefully minimize "That record declaration was right here last week!")

[ben-albrecht]

Resolving https://github.com/chapel-lang/chapel/issues/5456 may remove the need for some of these changes.