silnrsi / smith

font development, testing and release
Other
14 stars 5 forks source link

Add support for external tests folders #68

Open jvgaultney opened 2 years ago

jvgaultney commented 2 years ago

Please add support for running tests using test files stored outside the main project repo.

At a minimum, enable additional tests folders to be specified in the wscript using relative paths from the project root, as in:

EXTRATESTS = "../additional/test/folder;../another/one"

This would allow processing of test texts that cannot be included in a public repo for various reasons (e.g. license restrictions). The test results would be placed in the normal results folder.

An additional option would be to allow folders to also be specified on the command line. These would not override the EXTRATESTS specified in the wscript, but rather be run in addition to them.

jvgaultney commented 2 years ago

In our discussion there was some disagreement about whether the command line should completely override whatever is in the wscript (which would be simpler), but I think it would be better for it not to override. One option for making that simpler would be have two additional tests folder lists: EXTRATESTS defined in the wscript and MORETESTS as a command0line parameter. That would take care of the scenario where EXTRATESTS is something that should always be run and MORETESTS points to a folder of temporary ones only needed at the moment.

bobh0303 commented 2 years ago

I'm not sure we need to introduce a new wscript variable.

In the same way the wscript DOCDIR was expanded to handle a list of documentation directories, e.g.:

DOCDIR=['documentation', 'web']

Why not expand functionality of TESTDIR to take multiple test directories (some of which could be outside the project)? e.g.:

TESTDIR=['tests', '../additional/test/folder', '../another/one']
jvgaultney commented 2 years ago

Good idea! It's also good to be consistent about how wscript handles paths.

mhosken commented 2 years ago

But if you want an override-able default, then we would also need EXTRATESTDIR. So, I propose:

n7s commented 2 years ago

I can confirm TESTDIR and EXTRATESTDIR can now hold multiple path lists and test data in these multiple folders are taken into account by the smith tests targets.