l3build check reports misleading if no tests defined

[cfr42]

I'm not sure if this is a bug. Maybe it is a request for clearer reporting.

If I run l3build check with a test file which fails to define any tests and an expectation file which also fails to define any, then l3build reports as follows:

Running checks on
  mini-test (1/1)

  All checks passed

It is true that all checks passed. It is equally true that all checks failed. So the final claim is technically correct, albeit misleading. The (1/1) is also confusing, I think, at least if you've seen this kind of output from make, where similar tallies typically indicate the number of tests rather than the number of files.

testfiles/mini-test.lvt:

\input regression-test.tex\relax
\typeout{hylo}

testfiles/mini-test.lve:

\input regression-test.tex\relax
\typeout{hello}

Could you clarify the output somehow? Some reminder that the numbers mean files rather than tests, maybe, and/or No checks failed rather than All checks passed. Insofar as I understand the code, isn't the former rather than the latter what's actually being reported here?

Failing that, you could remind people that l3build allows an empty domain, but while this is entirely logical, it is not very intuitive. There is something quite odd about the fact it makes claims such as 'All flying pigs speak French' and 'Every unicorn loves a Womble' true.

[josephwright]

'All tests' from an l3build point of view means all .lvt files - that's the only number we can reasonably track as we can't really count up inside multiple .lvt files. As such, 'all pass' means that no comparisons fail: and that's true here as the .tlg file and the 'live' output match. We have tests that actually work that way - a .lvt file that produces nothing unless there is an issue. So I don't think we can change this - it's essentially by-design.

[josephwright]

Note that whilst many more recent .lvt files include at least one \TEST line, that's not required and not seen in many of the older ones at all. So counting 'tests' is tricky.

[davidcarlisle]

Your test is not empty and could fail if you changed the tlg. Use of regression-test.tex and \TEST.. are entirely optional (and not used in most of the tests in the latex2e repo, most of which predate l3build). See for example one of the most recent

https://github.com/latex3/latex2e/blob/develop/required/tools/testfiles/github-1378.lvt

[cfr42]

So what is the criterion for passing exactly? If compilation of the test file fails with a fatal error, I'm told all checks passed, so it doesn't seem to count by files, but it also doesn't count by \TESTs, so what does 'all' mean? All what?

[davidcarlisle]

the test file might have an error, but if that error is expected and you check in a reference tlg file with the same error outcome, then when the test runs the generated log and the reference tlg will be the same and the test will pass. Use of \TEST is entirely optional (most tests under latex2e do not use that at all) and l3build check acts the same way whether regression-test.tex or test2e.tex or neither is used.

[matteofg]

And what is test2e.tex? It doesn't seem to be available in any LaTeX distribution....

[muzimuzhi]

@matteofg

test2e.tex was developed along with latex2e so predates l3build by some decades. It is part of the latex sources but was never documented or distributed for general use.

see https://github.com/latex3/latex2e/pull/1013#issuecomment-1465037690 by @davidcarlisle for more info

[matteofg]

@muzimuzhi Ok, what I guessed... thanks

[cfr42]

@davidcarlisle But what is it actually looking at when it says 'All checks passed'?

If there's an unexpected compilation failure, I still get 'All checks passed'. This is with an .lve rather than a .tlg.

[FrankMittelbach]

@cfr42 I'm not sure I understand your scenario. The test directory is cleaned prior to making the tests. So if a compilation failure doesn't produce an output file at all, l3build should find a difference to the saved version of the output file, and thus reporting a failure. If that doesn't happen and instead the comparison fails in a way that it is not reporting this difference this this is a bug.

However, if your test compilation produces an error and you have previously certified this error as "this is the correct result of this test" then it is, of course, correct that no failure is reported. The same is true for your initial example above: if your test produces no output and you certified that as correct, then it is correct that this file is marked as passing its tests.

We do that quite often in the LaTeX test suite, e.g., write an lvt file that only shows output if a certain issue shows up, but is empty otherwise. We then certify the empty one. Basically, the process is: a) you write an .lvt and save its output b) you manually check that what it show is what you want to certify as being the correct result (which might be show no output, in contrast to, say, a TeX error like, \foo has an extra }. Then you stick that into your test suite and it reports success until the day you get different output.

I sometimes forget to have a \START in the right place and so my test output doesn't show what I intended it to show for comparison, but then it is my fault if I don't manually verify the result, but blindly add lvt and tlg into the test suite.

[josephwright]

'All checks passed' means that a run generates no .diff (or .fc) files - that running diff (or similar) for each test file doesn't yield a difference from the reference version. That does not rely on any specific structure to the tests, the outputs, etc.: as @FrankMittelbach says, we have tests that deliberately force 'failure'.

[davidcarlisle]

@davidcarlisle But what is it actually looking at when it says 'All checks passed'?

If there's an unexpected compilation failure, I still get 'All checks passed'. This is with an .lve rather than a .tlg.

the processing for lve is simimilar to that with tlg. You have checked in an lvt that produces some (normalised) log file, if you then check in a lve that produces the same normalised log, that test will pass. It doesn't matter how many \TEST the file has or whether or not it generates tex errors, a test passes if the two normalised logs are reported as having no diff.