improve testing support and the CI

[benoit-pierre]

Testing:

• fix testing on macOS: ensure the same temporary directory is used by redirecting all /tmp/… paths accesses to the same location (similarly to was is done on Windows)
• add testing support to the cmake build
• add support for running the tests in parallel: isolate each test temporary directory

CI:

• fix the cmake_win64 workflow: the libtiff build was failing (disable LZMA support for now), the version of giflib being built was 5.1.2 (broken)
• enable the cmake_win64 workflow for all branches
• stop a workflow previous run when a new one is started (on the same branch/tag)
• enable the tests on the cmake_… workflows

[DanBloomberg]

Can you split out the utils2.c "/tmp" rewriting for Apple from the rest of this PR? That's the only part of your PR for which I may have semi-intelligent input. For that, please add comments at the top of genPathname() for the Apple rewrites you are adding.

In one of your cmake changes, at line 41 I saw "sudo sudo ..."

I see that you're implementing the test suite in cmake. It also looks like you're allowing some degree of parallelism. I am vary interested in the results. In linux, if you allow too much parallelism the tests will fail due to race conditions between the run that generates the golden data and the one that compares with it; i.e., the generation run has not completed and the compare run overtakes it.

[DanBloomberg]

Just to be clear on the part about the race condition in the tests in linux: this is when reg_wrapper.sh is used with autotools, and you invoke the test run with make -jN check, with a value of N that probably should be less than half the number of cores to avoid the problem.

[benoit-pierre]

Can you split out the utils2.c "/tmp" rewriting for Apple from the rest of this PR? That's the only part of your PR for which I may have semi-intelligent input. For that, please add comments at the top of genPathname() for the Apple rewrites you are adding.

Yep, it's a separate commit, see #709.

In one of your cmake changes, at line 41 I saw "sudo sudo ..."

Amended.

I see that you're implementing the test suite in cmake. It also looks like you're allowing some degree of parallelism. I am vary interested in the results. In linux, if you allow too much parallelism the tests will fail due to race conditions between the run that generates the golden data and the one that compares with it; i.e., the generation run has not completed and the compare run overtakes it.

My analysis of the issue is that a test generate run can interfere with another test compare run.

[benoit-pierre]

Just to be clear on the part about the race condition in the tests in linux: this is when reg_wrapper.sh is used with autotools, and you invoke the test run with make -jN check, with a value of N that probably should be less than half the number of cores to avoid the problem.

I can run the testsuite successfully on Linux with make -j$(($(getconf _NPROCESSORS_ONLN)*2)) check or ctest --progress --output-on-failure --parallel $(($(getconf _NPROCESSORS_ONLN)*2)). In fact I can run both commands in parallel ;).

[DanBloomberg]

@benoit-pierre This is still open. PR 709/713 is in, and there are conflicts. If you resolve them we should be able to merge your changes. Thanks!

Dan

[DanBloomberg]

@stweil, @zdenop

This set of commits is a major advance in testing functionality of the library and programs on different platforms.

(1) Commit add support for running the tests in parallel improves my original 'logic' in genPathname()and additionally adds support for temp dir rewriting in linux. l This is something I'd initially decided against, but it seems like a reasonable option to support if someone wants to use it. These look fine to me. Stefan, do you agree?

(2) Commit cmake: add testing support has the major changes in the CmakeLists.txt files and reg_wrapper.sh. (3) The other commits are for workflows. These all look reasonable to me. However, they are also far above my understanding of good practice, and need to be reviewed/approved by someone who supports these builds.

Dan

[stweil]

There already exists a convention for rewriting the directory for temporary data by using the environment variable TMPDIR. If we want to support such rewriting for Leptonica, too, I'd support TMPDIR instead of introducing a new environment variable LEPT_TMPDIR.

One problem which must be solved for parallel tests is that some Leptonica functions use "temporary files" with filenames which cause a conflict if more than a single process uses such functions. That problem is a general problem. If it is solved for the tests by using many different temporary directories, it still remains for all other use cases. With the current pull request two different parallel runs of make check -j (for example with different compiler options) would still fail. Therefore I think that the approach here to solve parallel tests is not the right one.

Another problem of the tests is that some tests depend on others, so the order in which tests are run is important.

And finally a complete fix for parallel tests must also handle the autoconf builds (= fix prog/Makefile.am).

Unrelated: @DanBloomberg, why are those files in /tmp/lept not real temporary files which get removed after they were used?

[DanBloomberg]

> There already exists a convention for rewriting the directory for temporary data by using the environment variable `TMPDIR`. If we want to support such rewriting for Leptonica, too, I'd support `TMPDIR` instead of introducing a new environment variable `LEPT_TMPDIR`. Seems fine to me to use TMPDIR

> One problem which must be solved for parallel tests is that some Leptonica functions use "temporary files" with filenames which cause a conflict if more than a single process uses such functions. That problem is a general problem. If it is solved for the tests by using many different temporary directories, it still remains for all other use cases. With the current pull request two different parallel runs of `make check -j` (for example with different compiler options) would still fail. Therefore I think that the approach here to solve parallel tests is not the right one. Benoit-Pierre says he has run tests in parallel executions successfully, as well as using multiple cores on the same run. But even if there are some failures, I'm not particularly concerned about race conditions accessing temporary files for testing. In fact, I believe that your recent change making index variables atomic will help avoid race conditions.

> Another problem of the tests is that some tests depend on others, so the order in which tests are run is important. I fixed the only one I knew about, which was unearthed by this or another recent set of PRs

> And finally a complete fix for parallel tests must also handle the autoconf builds (= fix `prog/Makefile.am`). This may have been done with the changes made to reg_wrapper.sh, at least to the same extent as the tests with cmake

> Unrelated: @DanBloomberg, why are those files in /tmp/lept not real temporary files which get removed after they were used? So that they can act as true regression tests. I run the tests with generate to get the baseline values, before making any code changes. There are about 170MB of "golden" files in /tmp/lept/golden. After any changes are made, I rerun to check for failures. Each compare run puts files in /tmp/lept/regout, as well as many other subdirectories of /tmp/lept, and these are compared with the golden files. For diagnostic purposes they must not be erased. They are overwritten on the next compare run.

[stweil]

I just read the commit message for c72049d16d36ca21d8549f9ae8eefc4fdf5a1807. That reasoning is still valid, not only for UNIX / Linux, but also for other operating systems like macOS and Windows. So there is a Tesseract bug caused by Leptonica's rewriting of /tmp for Windows since a long time, and we now have introduced the same bug for macOS which is more severe because on macOS /tmp always exists.

I'm afraid I need more time to find a good solution for that mess.

[DanBloomberg]

Are you saying there is a current bug in Tesseract on Windows due to rewriting, or in old versions from more than 5 years ago?

As you can see from that commit and others around the same time when genPathname() was being cleaned up, I was trying to minimize rewriting because there is something conceptually ugly when a programmer says to write to a file 'x' and instead it quietly gets written to 'y'. At the time there seemed no reason to do it on Linux. Perhaps we should use Benoit-Pierre's version (LEPT_TMPDIR) to make such Linux rewriting a more deliberate decision than allowing use of a default?

Fixing Tesseract is top priority -- whatever you come up with is OK with me.

[DanBloomberg]

make -j does not limit the number of jobs, so I am not surprised there are race conditions between generation and comparison that cause failures.

I have no objection to removing the newly-added -j argument from the script, to avoid testing failures that will just upset people.