Imagemagick not producing diff files

[rpspringuel]

I've set up an Ubuntu 18.04 VM for the purposes of testing the TeX Live 2019 pretest and have come across a problem in the test suite. It seems that the diff files aren't being produced.

Originally, I was getting convert errors and tests were failing due to the inability to create the images to be compared. To fix this I disabled the security policies for ImageMagick (as described here). The convert errors went away and the comparison tests are happening properly (and appear to have the correct results).

However, it doesn't appear that the diff files are being created. When I review results (whether through the script or manually) I cannot find/open the diff files, though the expectation and results are both there. In this case the problems are clear enough that I don't need the diff to spot them, but we'll probably need to fix this before long.

Version: ImageMagick 6.9.7-4 Q16 x86_64 20170114 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP 
Delegates (built-in): bzlib djvu fftw fontconfig freetype jbig jng jpeg lcms lqr ltdl lzma openexr pangocairo png tiff wmf x xml zlib

[rpspringuel]

That "before long" is now here. The bug indicated in gregorio-project/gregorio#1446 has been fixed, but I'm still getting "results differ" test failures. Further, the differences are small so I can't spot them manually (at least for the tests I've actually looked at).

Is anyone else running Ubuntu 18.04 that can check to see if they see the same problem?

[rpspringuel]

Oddly, I just tried running the convert command responsible for generating the diff file directly from the command-line and it worked just fine. Any ideas as to why the command would fail silently in the test script but succeed directly from the command-line?

[rpspringuel]

Okay, I take back what I said above. The problem doesn't appear to be the convert command to create the diff files failing; that code is never getting executed. I'm testing the control logic right now to determine what is happening to prevent the code from getting into the loop where the convert command occurs.

[rpspringuel]

Found something. The expectation images aren't being found so if [ -f $expected ] returns false and bypasses even the difference check to declare a failure (the same failure as if the difference check had been done and actually failed). I'm going to submit a patch that differentiates the failure messages so it's easier to see this in the future.

Unfortunately I'm out of time for today, but I still need to explore why the expectation images aren't being found. There's an earlier failure message which is supposed to trigger if they can't be created. Since that message is not happening either something is going wrong so that it looks like they are being created when the reality is otherwise, or they are somehow disappearing between the creation and the later test I indicated above.

[rpspringuel]

Found the problem. During the first run of gregorio-test.sh the image-cache folders were created like normal, but because of the restrictions in place on convert at the time, these folders were not populated with any images. Then, on subsequent runs gregorio-test.sh saw that the image-cache folder existed and was newer than the corresponding expectation pdf and so did not try to create the images again. This is what caused if [ -f "$expected" ] to return false and skip the comparison. Once I realized that the image-cache folders were empty, I ran gregorio-test.sh -i to force the rebuilding of the cache as the tests were run and things worked out fine. When I edit the failure messages I'll keep this in mind so that they're more informative.

[rpspringuel]

See #330.