search

apertium / apertium-regtest

Regression testing system for Apertium language data and translators
https://wiki.apertium.org/wiki/Apertium-regtest
GNU General Public License v3.0
1 stars 0 forks source link

total count of gold matches in test mode #28

Open jonorthwash opened 1 year ago

jonorthwash commented 1 year ago

Currently apertium-regtest test has output like this:

$ apertium-regtest -c .-morph test
Corpus 1 of 3: -s plurals-morph
  13/13 (100.0%) tests pass (3/13 (23.08%) match gold)

Corpus 2 of 3: -es plurals-morph
  20/20 (100.0%) tests pass (3/20 (15.0%) match gold)

Corpus 3 of 3: irregular plurals-morph
  22/22 (100.0%) tests pass (2/22 (9.09%) match gold)

All tests pass.

It would be nice if the bottom line (or somewhere around there) said something like "8/55 tests match gold" or similar.

jonorthwash commented 1 year ago

Also, the wording "all tests pass" is misleading when so many don't match gold. Perhaps that wording should be clarified too.

TinoDidriksen commented 1 year ago

"All tests pass" is correct. Gold is not relevant to test passing - gold is just an extra step to keep track of "perfect" translations, but has nothing to do with whether the translation changed. This is not a gold tester, and must not be a gold tester. It's an important distinction and different mindset.

jonorthwash commented 1 year ago

This is not a gold tester, and must not be a gold tester.

Okay, sure, but for morphology this will be the primary use, especially for those of us using it as a replacement for aq-morphtest and morhp-test.py. So, okay, leave the terminology and mindset as is, but we could still use some minor additional features that resemble gold testing to make it more useful for morphology work.