Convert chpldoc tests to match against generated .rst

[lydia-duncan]

Though we lean more strongly on the .rst and .html output generated from chpldoc through Sphinx, our testing of chpldoc still compares against the old .txt output for the most part. It would be nice to update most of the expected tests to match against .rst (which is nicer to look at than .html), with a couple to verify the .txt output so long as we maintain it. Unfortunately, this work is pretty thankless, which is why we've put it off so long.

This impacts the tests in the test/chpldoc directory

[bradcray]

Unfortunately, this work is pretty thankless

I would give hearty thanks to whoever completed it! In particular, I think it's problematic that we do nothing to verify the consistency of our generated documentation other than hope that we'll notice when something about it goes wrong. By having the chpldoc tests verify the output from which our real docs is generated, we'll have far greater confidence that we aren't breaking chpldoc as we improve the compiler.

Once this is done, I'd advocate for retiring our homegrown .txt mode, just to reduce the amount of code that we carry around and don't really benefit from (that I can tell).

[lydia-duncan]

Sorry, by "thankless", I meant "dull and grinding". I agree with what you said above

[bradcray]

Is the work itself hard, or just the update of all the tests? Assuming the work doesn't present a challenge, what if we just implemented it, ran the test system, bulk-copied all .exec.out.tmp files into their .good equivalents, spot-checked a few of them to make sure they were reasonable, and then accepted the whole batch just to put a stake in the ground in order to start catching inadvertent slips?

[lydia-duncan]

The change could be easily scripted. One would need to update all the directory level CHPLDOCOPTS files to throw the rst flag instead of the --text-only flag, and all the .catfiles to look in the appropriate place for output.

Left to my own devices, I would prefer to take this on a directory-by-directory approach, to ensure correctness was maintained over time. It would take longer than assuming things outside of the spot check were fine, but it would be less error-prone, especially considering there isn't a big "test o' all the features". And it would be less likely that we'd assume things were fine and never look at the expected output again. Thoughts?

[bradcray]

I'd rather live with un-reviewed .good files that approximate what we're generating for the real docs than to continue using the threat of a manual review to continue putting this off and hoping that the .rst files don't break in the meantime. I'm also not convinced I'd want any of our developers to spend their time putting in that slog given other priorities. My assumption is that snapshotting the current behavior would draw a line in the sand and that we'd be aware of backslides from that point on (and would become aware of problems as we continue to develop and review chpldoc rst->html output for the modules).

I'd been imagining we'd use a rst->txt capability to continue comparing text files to .goods (or reimplement the --text-only flag to run the rst->txt step?), but I'm also open to testing the .rst directly as well, and maybe that's best if it's not particularly difficult for humans to read.

[lydia-duncan]

Okay, we'll follow that path. Is it alright if I open an issue to track that the .good files haven't been verified?

[bradcray]

Fine with me.