josh-hadley
commented
2 years ago To expand on @markusicu's comment, the fontbakery test ("check") framework has a space for each check to define a "rationale" which is a more-or-less plain English description of what the test is checking and why. Additionally, failure messages are plain English and typically describe expected versus actual values and a description of why the test failed. A good example of both a rationale and descriptive failure messages is here: https://github.com/googlefonts/fontbakery/blob/15cff7721773dbb6a32997fc22359386e80e6830/Lib/fontbakery/profiles/head.py#L39-L79 It would be really helpful if we could somehow adapt the invariant test scheme to use some of these ideas (for example, perhaps capture a preceding comment as a 'rationale' that is output as part of a failure message, and have some way to express expected data where applicable, pointers as to what might cause a failure, etc.). Ideally we'd have a scheme that allows us to make adaptations incrementally so we can do continuous improvement.
macchiati
TestUnicodeInvariants needs a fair bit of experience to make sense of its output.
Aside from running it automatically, the text file should have more documentation for the rationale for each test, and for how to fix failures; e.g., how to decide which properties to change, whether to add to an exceptions list, or whether to modify the "invariant" altogether. (Is it set in stone via a stability policy? Is it just based on someone's mental model of how ideally properties should correlate?)
It should probably be a parameterized JUnit test, parsing each test condition into a separate test case. When a test case fails, it should print the relevant documentation comments. The CI bot output should include enough information to start tackling the test failure; it should not merely say that TestUnicodeInvariants as a whole failed.
UnicodeInvariantTest.txt might need more structure so that the test code can easily group comments and test conditions. Currently, there are sometimes several comment blocks before a series of both variable definitions (setup) and test conditions.
@josh-hadley mentioned that https://github.com/googlefonts/fontbakery has good test failure output as suggested above (or better :-).
See the 2022-may-10 PAG meeting minutes for context.