Open roberth opened 6 months ago
Did a small amount of research:
Functional tests
Performance matters, so we want to log efficiently.
When logging is enabled, do not create child processes (ie use echo
/printf
).
Do not reopen the report file, but use a file descriptor. Set CLOEXEC, e.g.: exec 3>&-
.
Report format
JUnit XML looks suitable. Open questions:
testsuite
s or flattened with .
? This is mostly isomorphic, so it probably depends on the tooling we use, and even then conversion is possible, esp. flattening which is lossless. This suggests nested should be the default layout, and we could preprocessing before certain tooling needs it, but maybe that's not good DX.properties
for representing the test environment (from the test matrix)? Alternatively, we could have a massive hierarchy in the test suite names, but that does not seem conducive to analysis, because it doesn't convey the kind of equivalence that a duplicated test case with different properties
has. Seems like properties
is better, but we may have to revisit this.Tooling
Is your feature request related to a problem? Please describe.
Nix runs in many environments, each of which needs testing. Not all tests run in all environments. Any non-trivial logic in test code can contain bugs, which will go unnoticed. Test conditions that control whether a test runs in particular are fail unsafe. (E.g.
if system == "x86-64_linux" ...
. Did you spot it?)Given the source of any test, I want to be able to tell reliably,
Describe the solution you'd like
Make all tests report whether they've run (esp. in case of the Nix package), and in which situations they've run in case of a test matrix.
Write a derivation that aggregates all the test reports from all our builds (
packages
,checks
,hydraJobs
, etc)x86_64-linux
withkvm
.Make all derivations report which tests exists. Join this to the test run reports. If not a single test run report is found, fail.
Describe alternatives you've considered
Just hope.
Use an existing solution? Didn't find one. Most of our test code is custom, so I have doubts, but if anyone knows anything, please comment. JUnit's report format is a standard that's exported by a number of frameworks. Would be nice to reuse that perhaps. Maybe combine this with a cross referencing solution? Something like GHC Notes, or something that formalizes "grep across files with identifiers". Combine with coverage report? Combine the coverage reports' per-line info? (Where we have coverage reporting; something else elsewhere)
Additional context
Priorities
Add :+1: to issues you find important.