Create a file with commands and expected result (i.e. log messages). Check which log sections are there (i.e. which engines/extensions ran) and how many (which?) messages of which type are included.
Specify tests in a structured way. (Ruby? JSON with schema?)
which test document (if any)
which parameters (potentially multiple sets?)
the expected behaviour: which engines and extensions are run how often and when?
the expected output: number of errors and warnings; optionally their content
Run (subsets of) the tests.
Copy test document to separate directory.
Run multiple tests in parallel.
Compare with expectations
Aggregate results and output a report (options for machine and human readability).
Bonus: Can we script/wrap the resulting test bank so that it can be automatically run by some CI tool? That would be helpful evaluate pull requests (if we should get any).
We need some form of automated test.
Create a file with commands and expected result (i.e. log messages). Check which log sections are there (i.e. which engines/extensions ran) and how many (which?) messages of which type are included.
Depends on #103