Move all tests from test/ subdir to test/subdir in the component being tested

[flammie]

Reorganising followingly:

• [x] test/X/Y -> src/X/Y/test
• [x] all make checks should pass
• [x] fix some tests that don't work with parallel-tests
• [x] move test/run-yaml-testcases.sh.in and test/run-morph-tester.sh.in to giella-core. \ This requires that language-specific parts be moved to other files, at least the enabled fst tech identification. It also requires updating the fst detection algorithm - some tests fail now because they can't find the required FST.
• [x] also factor out common program logic from src/fst/morphology/test/generate-*

[snomos]

There are a number of files still in the old dir structure, e.g.:

lang-est-x-plamk/test/tools/
└── spellcheckers
    ├── Makefile.am
    └── fstbased
        └── desktop
            └── hfst
                └── accept-all-lemmas.sh.in

lang-sma/test/tools/
├── mt
│   └── apertium
│       └── Makefile.am
└── spellcheckers
    ├── Makefile.am
    ├── fstbased
    │   └── desktop
    │       └── hfst
    │           └── accept-all-lemmas.sh.in
    ├── listbased
    │   └── plx
    │       ├── expected.plx
    │       ├── words-accept.exp-out.txt
    │       ├── words-accept.txt
    │       ├── words-reject.exp-out.txt
    │       ├── words-reject.txt
    │       ├── words.regex
    │       └── words.txt
    └── speller-testbed-sma.txt

Must these be cleaned manually, or are they already copied to the new location, so that they can just be deleted?

[flammie]

Some are language specific tests that need to be ported still, test/tools/spellcheckers/listbases/plx for example. Perhaps accept-all-lemmas is something that could be in template too. And for few fiels like test/tools/spellcheckers/Makefile.am that are copies of tools/spellcheckers/test/Makefile.am can be deleted, not sure why scripts didn't work there.

[snomos]

About listbased/: since we do not build such spellers anymore, the data does not need to be moved to the new test dir locations. Instead I suggest we move the data to test/data/.

I also suggest that we rename test/data/ to testdata/. The idea is that this dir contains testdata used by several other tests, ie is shared among tests in various locations. Test data that is specific to a (set of) test(s) in one dir should stay with the tests.

[snomos]

Things failing after the re-org:

• [x] https://github.com/giellalt/lang-sma/issues/100 (failing only in bash, including CI, working in zsh)
• [x] We do not get analysed and generated missing lemmas in our editor automatically, as before. The files src/fst/morphology/test/generated_missing_adjectives_lemmas.hfst.txt and ...analysed... exist, but are not shown automatically, as before.
  • we now have a new target for that usage: make devtest. It will run all tests the old way, but is slower. make check now uses a parallel test runner, so you can even run tests with make check -j, running all in parallel. It is thus much faster, but the test summary is different.
• [x] yaml testing report were earlier shown on a file-for-fileb basis, this now does not happen, instead, we get an overall summary.
  • see previous item. This is how it is supposed to work now, use make devtest for the old behaviour
• [x] We earlier had a shellscript giving yaml tests directly , in e.g. lang-smn/test/yaml-check.sh (cf. git), this now does not work ( I tried both to change the path and to move the script to the new test catalogue, nothing helped)
• [ ] make check stops hanging after PASS: tag_test.sh(tested on vep and smn, both 25 min and counting). This is not a general problem, as fit gets past this point immediately. nob hangs on the next command (adjectives), there is thus sand in this machinery.
  • make check hangs on open files, thus if make check finds lemma errors, and open error output files with SubEthaEdit (see), make waits for the opened file to be closed before proceeding. The solution is to open files using see <FILE> &, so that make can continue without waiting for the file to be closed. I have tested this, but not yet come around to roll out the necessary changes to all languages.

[flammie]

I think some of the issues highlight that make check is not ideal for the purpose it was used, its main purpose in life is just to act as a litmus test for automation and external users that the downloaded version works within specified parametres (hence make -j && make -j check && make install). The tooling to support developers (e.g. opening log files in an editor) should probably be decoupled to devtools/ or devtest.

One way to use the current test driver efficiently is to keep test-suite.log open in an editor that supports automatic reloading, it should contain most of the stuff previous test driver outputted on terminal (for failing tests).

[flammie]

the make check uses now see & if giella-core-1.0.1 is configured to find one, but I haven't tested it, on linux etc. it falls back to head which is quite fine for logs