[ ] comprehensibly document UI testing in the currently empty README.adoc
[ ] introduce a format option -f/--format which accepts pretty or terse (taken from rust's libtest) where the terse format prints single tests single characters without space and line breaks (. for passing tests, i for ignored ones, g for gilded ones, I for invalid ones, a for auxiliary ones (not shown by default)) and maybe even make the terse the default
[ ] syntactic or preferably semantic highlighting for test configurations by the VS Code extension
[ ] improve the looks of the generated diffs by diff'ing diff'ed lines by character (only those lines where it makes sense, I think that's single + lines preceded by single - lines) and highlight them (changing their background color)
[ ] check how much the current architecture of auxiliary files impacts the (parallel) performance of the test runner. I suspect since we are skipping so many files, every thread wastes a lot of time and performance is not as good. basically the division of work is suboptimal. also for packages, we heavily rely on auxiliary files and the more package tests we add the more the performance will probably degrade. could the auxiliary/ folder design from rustc's test suites help us out?
[ ] revision / matrix system
[ ] support putting the test configuration into a separate file named <testname>.testconfig which allows us to test files where we don't want to have any comments e.g. to test everythings ok with truly empty files or lexing and spans are fine without comments when lexing (…), you get what I mean
[ ] smh. (somewhere) record the modification date of passing tests and skip them (printing ignored or ideally smth. more descriptive) when they are tested and if they were indirectly referenced (e.g. through a folder)
[x] use proper lexers to obtain the comments (to get rid of false-postive test config params) (via #162)
[x] ensure that the unit tests of the testing framework are executed in CI (haven't checked yet)
[ ] check if the framework actually works on Windows (we currently use the POSIX? timeout command for which I am not sure if it is available)
[ ] add a flag to test all git-unstaged or all git-staged test files
[ ] skip / ignore files that have already been tested & passed: record modification time / hash of test files & the compiler; if any of them differ from the current run, don't skip; this might not be very useful right now; add a flag to force-run all tests
Split from #140.
README.adoc-f/--formatwhich acceptsprettyorterse(taken from rust'slibtest) where the terse format prints single tests single characters without space and line breaks (.for passing tests,ifor ignored ones,gfor gilded ones,Ifor invalid ones,afor auxiliary ones (not shown by default)) and maybe even make the terse the default+lines preceded by single-lines) and highlight them (changing their background color)auxiliary/folder design from rustc's test suites help us out?<testname>.testconfigwhich allows us to test files where we don't want to have any comments e.g. to test everythings ok with truly empty files or lexing and spans are fine without comments when lexing (…), you get what I meanignoredor ideally smth. more descriptive) when they are tested and if they were indirectly referenced (e.g. through a folder)timeoutcommand for which I am not sure if it is available)