Approve behavioural changes by some "oracle"

[maxbechtold]

This is related to #2 in that it allows approval on-the-fly, but in a more elaborated manner.

E.g. there's a use case to adjust a program while it is only covered by golden master tests. This could be as simple as adapting the output format, or more complex changes. For very regular changes (e.g replace space by tab) there could be an automatic way to approve changes, if the output changes in the described way. This could leverage the power of tools like kdiff3 and sed.

Questions:

• is it enough to let the user provide an "oracle" method like assess(input, output)?
• how to ensure this "oracle" is only active once/for the current change?
• how does this relate to the comparison built into FastApproval (cf. ExecutableDifferenceReporter)?

[maxbechtold]

I did a rough sketch of using native programs as oracles on the branch oracle-issue18.

The ExampleGoldenMasterTest shows how a ExecutableDifferenceReporter can be used as an oracle to judge differences, where it compares text disregarding case (using Windows' FC). It uses a locked oracle that applies once, then generates a .lock file. With the .lock file in place, the oracle is not asked (avoiding accidental approval of mismatching outputs). The user can reenable the oracle (without changing code) by removing the .lock file.

Similarly, one can define an oracle that is always asked to approve mismatching outputs, e.g. one that treats "__" and \t as equivalent (i.e. accepts tab for 2 consecutive spaces and v.v.).

[maxbechtold]

Also, one could provide a method for comparison of master files and output files instead of falling back to an assess method.

Objects serialized to JSON is also a use case where there's a lot of textual flexibility that should probably not lead to failing master tests.

Thanks to @almondtools for the suggestion.