Maybe: Plunder golden inputs for TEST-PRINT-PARSED-PROGRAM-GOLDEN-FILES

[appleby]

As discussed in #257, it might be useful to have a function to steal golden inputs from other Quil test files for use in test-print-parsed-program-golden-files.

Two potential sources of pilfered golden inputs are tests/good-test-files/ and tests/compiler-hook-test-files/. Of the two, good-test-files appears to have the juicier inputs. Possibly there are other sources that could be plundered as well.

It's debatable whether this is a good idea or overkill, and probably depends on how often new test cases are added to the above directories. Note, however, that plundering golden inputs has already yielded one potential bug (#272).

Additional considerations

Any such function would probably need to maintain a blacklist of input files that should be skipped.

One reason to skip input files is that their printed form contains floats with lots of digits in the printed representation, which may or may not be portable and in any case seems sketchy to depend on the default formatting for ~F.

Also consider that INCLUDES would need to have their paths re-written since the newly produced golden file will not live in the same directory as the source file that is being plundered.

For example, the file tests/good-test-files/good-include.quil contains the following Quil program:

# include a file

INCLUDE "bell.quil"

BELL

In order to work with the existing test-print-parsed-program-golden-files test, the INCLUDE path would need to be re-written to INCLUDE "good-test-files/bell.quil", and also the test would need to be run with *allow-unresolved-applications* bound to t (I think).

Alternatively, any file containing an INCLUDE could be added to the blacklist.

POC bash script

Here is a proof-of-concept bash script. If we decide to implement the gold-plundering mechanism described in this issue, then the script would be converted to a lisp function and moved into tests/printer-tests.lisp alongside the function update-print-parsed-program-golden-files. Presumably, the cutesy pirate references would also be toned down!

This script is written with the assumption that it lives in the directory tests/printer-test-files/.

#!/usr/bin/bash

BURIED_TREASURE="../good-test-files/*.quil ../compiler-hook-test-files/*.quil"
TREASURE_CHEST=plundered-gold.quil

for gold in $BURIED_TREASURE; do
    echo "# Input"
    echo "# Plundered from $gold"
    cat "$gold"
    echo
    echo "# Output"
    echo
done >> "$TREASURE_CHEST"

echo "Arr. All the gold has been plundered from $BURIED_TREASURE."
echo "Don't forget to run CL-QUIL-TESTS::UPDATE-PRINT-PARSED-PROGRAM-GOLDEN-FILES in your lisp REPL."

[appleby]

With the benefit of time, I'm now fairly sure that, despite the amazing pirate-themed puns I'll be leaving on the table, automating the process of plundering golden inputs is probably overkill.

I do still think it makes sense to do a one-time import of at least the test cases from good-test-files, since doing so uncovered a potential bug in #272, in which case #272 should probably be fixed first.