undisbeliever
commented
5 years ago If you don't mind, I could probably write a simple test suite for you. If your happy with this design, please let me know and I get to work on it sometime during the weekend.
We need to have three types of tests; error tests, block tests and assert tests.
This is an error test I use for the untech-engine memory-management macros. It consists of a // ERROR tag that tells a python script where the compiler should fail. The test passes if the compiler does not return EXIT_SUCCESS and correctly displays the failed line number.
It would not be very difficult to modify the script to work with wiz.
// Try and create two Data blocks that overlap each other
define MEMORY_MAP = HIROM
define ROM_SIZE = 1
include "../../../src/common/memory.inc"
createDataBlock(rom0, 0xc00000, 0xc000ff)
createDataBlock(rom1, 0xc00100, 0xc001ff)
createDataBlock(rom2, 0xc00100, 0xc0017f) // ERROR
createDataBlock(rom3, 0xc10000, 0xc1ffff)The block tests could be created by a similar design, a hexdump snippet of the expected output could be added to the top of the test file. The script would invoke wiz with the correct platform (or platforms if the code could work on multiple systems) and test if a subset of the output matches.
Only a subset of the output would be tested. There is no need to test the ROM header as it could change between wiz releases.
For example:
// PLATFORM sfc
// OUTPUT 000000 c2 20 1a 3a e2 20 1a 3a 80 fe
bank code @ 0x808000 : [constdata; 0x8000];
in code {
const @ 0x80FFFC = reset; // emu reset
#[fallthrough]
func reset() {
mem16();
#[mem16] {
aa++;
aa--;
}
mem8();
#[mem8] {
a++;
a--;
}
while true { };
}
}The third type of test is an assert test and would be used to verify memory addressing and expressions are valid.
These tests would require an assert statement to be added to wiz.
Bananattack
As more development work happens on Wiz, it will eventually become useful to have an automated test suite that can ensure that no regressions have been introduced, and to ensure new features meet their requirements and work as documented. Also, as bugs are reported, it would be possible to add new tests to cover any problems that get fixed.
Being just one core developer, writing test harnesses can take a lot of time, and this is why it hasn't happened yet. However, I see the value in automated unit tests of various systems in the compiler, and functional tests to ensure the compiler produces the correct outputs given some inputs.
Being able to test that the compiler works as intended as it is developed will save time spotting regressions and debugging problems, and makes validation of internal commits and outside contributions easier.
Ideally it'd be cool to have a folder of test programs that could assert an exact output binary gets generated as a result.
Also, a way to validate type-system / intermediate detail stuff within the compiler.