Closed rpspringuel closed 5 years ago
That "before long" is now here. The bug indicated in gregorio-project/gregorio#1446 has been fixed, but I'm still getting "results differ" test failures. Further, the differences are small so I can't spot them manually (at least for the tests I've actually looked at).
Is anyone else running Ubuntu 18.04 that can check to see if they see the same problem?
Oddly, I just tried running the convert
command responsible for generating the diff file directly from the command-line and it worked just fine. Any ideas as to why the command would fail silently in the test script but succeed directly from the command-line?
Okay, I take back what I said above. The problem doesn't appear to be the convert command to create the diff files failing; that code is never getting executed. I'm testing the control logic right now to determine what is happening to prevent the code from getting into the loop where the convert
command occurs.
Found something. The expectation images aren't being found so if [ -f $expected ]
returns false and bypasses even the difference check to declare a failure (the same failure as if the difference check had been done and actually failed). I'm going to submit a patch that differentiates the failure messages so it's easier to see this in the future.
Unfortunately I'm out of time for today, but I still need to explore why the expectation images aren't being found. There's an earlier failure message which is supposed to trigger if they can't be created. Since that message is not happening either something is going wrong so that it looks like they are being created when the reality is otherwise, or they are somehow disappearing between the creation and the later test I indicated above.
Found the problem. During the first run of gregorio-test.sh
the image-cache folders were created like normal, but because of the restrictions in place on convert
at the time, these folders were not populated with any images. Then, on subsequent runs gregorio-test.sh
saw that the image-cache folder existed and was newer than the corresponding expectation pdf and so did not try to create the images again. This is what caused if [ -f "$expected" ]
to return false and skip the comparison. Once I realized that the image-cache folders were empty, I ran gregorio-test.sh -i
to force the rebuilding of the cache as the tests were run and things worked out fine. When I edit the failure messages I'll keep this in mind so that they're more informative.
See #330.
I've set up an Ubuntu 18.04 VM for the purposes of testing the TeX Live 2019 pretest and have come across a problem in the test suite. It seems that the diff files aren't being produced.
Originally, I was getting convert errors and tests were failing due to the inability to create the images to be compared. To fix this I disabled the security policies for ImageMagick (as described here). The convert errors went away and the comparison tests are happening properly (and appear to have the correct results).
However, it doesn't appear that the diff files are being created. When I review results (whether through the script or manually) I cannot find/open the diff files, though the expectation and results are both there. In this case the problems are clear enough that I don't need the diff to spot them, but we'll probably need to fix this before long.