Every TestSuite should have its score determined by weighting the scores of the component tests. Currently, the weights are all equal, so the total TestSuite score is simply the mean of the sort_key attributes of the component tests.
The attribute should be called weights and there should be possible to pass it as an argument to the constructor or to change it at anytime. It should also be possible to change it after a call to judge and then call judge again and immediately get a re-weighted score without running the tests again, using some sort of caching.
The weights (w) should be an 1-D array or a 2-D array. If it is a 1-D array, the calculation of the total score is simply s . w, where s is the array of score_key attributes, and w is an array of weights. w must be normalized to have a sum of 1 before it is set. If it is a matrix, the calculation is sum(w x s). For example, if w is the identity matrix (divided by the number of tests), then this is the same as equal weighting. However, if any off-diagonal entries are non-zero, then this means that some tests influence multiple entries in the resulting vector (w x s). This may matter in some implementations (see #99) that use this vector to do a non-dominated sort of the resulting test suite scores, retaining the whole vector instead of collapsing it into a single value.
rgerkin
commented
7 years ago
Every
TestSuiteshould have its score determined by weighting the scores of the component tests. Currently, the weights are all equal, so the totalTestSuitescore is simply the mean of thesort_keyattributes of the component tests.The attribute should be called
weightsand there should be possible to pass it as an argument to the constructor or to change it at anytime. It should also be possible to change it after a call tojudgeand then calljudgeagain and immediately get a re-weighted score without running the tests again, using some sort of caching.The weights (
w) should be an 1-D array or a 2-D array. If it is a 1-D array, the calculation of the total score is simplys . w, wheresis the array ofscore_keyattributes, andwis an array of weights.wmust be normalized to have a sum of 1 before it is set. If it is a matrix, the calculation issum(w x s). For example, ifwis the identity matrix (divided by the number of tests), then this is the same as equal weighting. However, if any off-diagonal entries are non-zero, then this means that some tests influence multiple entries in the resulting vector (w x s). This may matter in some implementations (see #99) that use this vector to do a non-dominated sort of the resulting test suite scores, retaining the whole vector instead of collapsing it into a single value.