a-v-medvedev
commented
1 year ago In branch 'output_rework' we try to address some of the issues mentioned here: 1) we have type-aware checks. The types are: float, int, string and bool. In basic case, the type is deduced from test_items.yaml contents, and chacks are made accordingly. For float checks, we use the value 'tolerance_float' as a default tolerance value. For int checks, the tolerance_int is used. Both tolerance_float and tolerance_int can be now set up on two levels: in common_params section to cover the whole test_items.yaml as default values; in each workload/workpart entry to cover defaults for a specific workload/workpart pair. One can also set up tolerance explicitely (overriding any defaults) for each section/parameter entry. The syntax for this setting is given below. For string and bool types, there is no such thing like tolerance: the comparison is always strict.
2) we have relative tolerance checks for float type. The parameter tolerance_kind is given the same way as tolerance_float and tolerance_int are given. This parameter can take one of values: 'relative' or 'absolute'. This picks up the way to handle tolerance_float.
3) explicit parameters choice for a specific section/parameter pair. Now besides a simple setting providing just a gold value to compare with, we can provide a descriptor that helps to set up precise comparison parameters. The descriptor in a YAML sequence with keywords and accomanying values that set up the comparator type, its options and the gold values, if relevant. For simple cases of float type comparisons, the setting looks like: [ absolute, GOLD_VAL, TOLERANCE_VAL ] or [ relative, GOLD_VAL, TOLERANCE_VAL ]. This setting overrides any defaults that are set up on higher level and set up everything explicitely for a specific section/parameter pair. The same syntax is available for int value type, the only requirement is that both GOLD_VAL and TOLERANCE_VAL must be integer.
For string and bool types, the setting [ equal, GOLD_VAL ] is meaningful.
The additional setting: [ oneof, VAL1, VAL2, ... ] is relevant for string and int types. It allows to point out the set of possible values for a variable that meant to be correct.
4) A specific pseudo-comparator is 'auxvalue'. The syntax is: '[ auxvalue, TYPE, AVERAGING_WAY, RECORD_NAME ]' where TYPE can be any of 'float', 'int', 'bool' and 'string' type descriptors. For this parameter no golden value comparison is meant, we do only averaging of the value over several test run attempts and record the result in output in the Auxvalues dictionary under the name NAME. The only averaging way that is supported currently is 'median'. Therefore TYPE that is meaningful currently is either float of int.
'auxvalue' pseudo-comparator allows one to collect some observed run-to-run data besides or in combination with strict functional testing. That is the way to easily collect productivity data at the same time as testing the application for crashes.
The Auxvalues disctionary mentioned above is a part or an entry in output.yaml for the testcases where the auxvalues are actually collected (if no such values present in a testcase, the dictionary is omitted).
We currently have only a hard coded way to check the parameters: each parameter can be checked to meet a given tolerance. We need much more flexebility: 1) relative tolerance checks; 2) checks of several params and their relationship to each other with some arbitrary equation representing the relations; 3) type-aware checks (like: integer values checks, bool values checks); 4) checks that contol values lay in an asymmetric diapason; etc.