Open gouttegd opened 1 month ago
Two things to do here:
First, for programs that are already part of the ODK: we need an inventory of them, so that we can add them to the existing tests.
Second, I’d like for the ODK project to adopt the following policy: whenever someone creates a PR that (among other things) adds a new program to the ODK, the PR must include a basic test that the program can at least run. (With the exception of programs that are provided by the Ubuntu distribution and that are installed by apt-get
: for those, it’s fine to trust the Debian/Ubuntu package developers.)
We’ve just had a case (as part of #1076) where one of the programs provided by the ODK (
j2
) was broken to the point that it could not even run. That program is not used in any of the standard ODK workflows, so it is not tested as part of the test suite and the issue was only caught after @matentzn did some extra tests on one of the ontologies he is involved with.While it may not be realistically feasible to include in the test suite some representative examples of the diversity of custom workflows that are used out there (and this is why the kind of extra testing we do on real-world ontologies will always be necessary), we should at least try to ensure that, if a program is provided in the ODK image, it can at least run without crashing or erroring out immediately.
There is already a basic infrastructure in place to test running some of the most important ODK programs:
This does not guarantee that the programs are fully functional, but it can at least catch basic issues like:
j2
);We should generalise the tests above for most of, if not all, the programs provided by the ODK, regardless of whether they are used in the standard workflows (especially if they are not used in the standard workflows).