search

gregorio-project / gregorio-test

A repository of tests for Gregorio
GNU General Public License v3.0
5 stars 3 forks source link

Sensitivity Selection #324

Closed rpspringuel closed 5 years ago

rpspringuel commented 6 years ago

The current method for adjusting the sensitivity of the tests is to change the PDF_DENSITY variable in gregorio-test.rc. However, in looking at gregorio-project/gregorio#1412 I came some information which may indicate that this is not the best way to adjust the sensitivity of the tests:

Using gabc-output/glyphs/broken-shapes as the trial case (with the old, and wrong expectation), I get the following results from the compare test at various settings for PDF_DENSITY:

150    0.998746
300    0.998631
600    0.998606
900    0.998601
1200   0.998575
1500   0.998549
1800   0.998559

This shows that while in general increasing the PDF_DENSITY will result in a lower (i.e. closer to failure) result, not all increases in its value are created equal (the change from 600 to 900 barely moves the metric while the change from 150 to 300 is almost 20 times larger) and some are even counter-productive (the change from 1500 to 1800 goes away from failure, not towards it).

Now, since PDF_DENSITY also affects the speed of the tests (compare takes much longer at the higher values) I do not think we should eliminate this parameter, but instead should look at a way to adjust the comparison that controls pass/fail determination more directly and in a user settable way. I need to do some more research, but my first results seem to indicate that using bc for the test would allow one to compare $metric to $THRESHOLD as floating point (instead of using the current coercion which allows bash's integer comparison operator to do the job), thereby making it easy to set $THRESHOLD in gregorio-test.rc

rpspringuel commented 6 years ago

See #325