It would be better if we had a different format for expected outcomes of the test cases instead of just plan text files with stdout.
Something like a custom file format that contains:
Expected returncode
Expected stdout
Expected stderr
This will simplify recording and replaying test cases and reduce the amount of required flags.
We could use something like JSON, but in a long term I plan to rewrite test.py in Porth too, so it has to be something that is easy to parse even in such a spartan language as Porth.
I'm thinking about a simple binary format:
|1 byte -- expected return code|
|8 bytes -- length of stdout|
|len(stdout) bytes -- the expected stdout encoded as UTF-8|
|8 bytes -- length of stderr|
|len(stderr) bytes -- the expected stderr encoded as UTF-8|
Such format is easy to produce/parse in both Porth and Python (using the bytes).
Using binary format will also enable us to assert binary outputs of the test programs. For instances, PPM pictures.
It would be better if we had a different format for expected outcomes of the test cases instead of just plan text files with stdout.
Something like a custom file format that contains:
This will simplify recording and replaying test cases and reduce the amount of required flags.
We could use something like JSON, but in a long term I plan to rewrite
test.py
in Porth too, so it has to be something that is easy to parse even in such a spartan language as Porth.I'm thinking about a simple binary format:
Such format is easy to produce/parse in both Porth and Python (using the
bytes
).Using binary format will also enable us to assert binary outputs of the test programs. For instances, PPM pictures.