Results to make difference need to change test status depending if are passing or failing. At first idea was to force clients to adjust test status. This commit adds possibility to Argus to track best results for given test and validate incoming results based on rules defined by Argus clients.
List of improvements:
Argus tracks best result for each column&row pair in tables, including run_id and result_date for future reference. Evaluated only when higher_is_better column metadata is set. Stored in separate table to track the changes (and correctly show validation rules calculated limits for given time)
Argus client may provide validation rules for each column. Supported validations (can set multiple of them):
fixed_value - verifying if value is better than it
best_pct - verifying if value is better than margin percent from best value
best_abs - vefifying if value is better than margin absolute from best value Validation rules are tracked for future reference.
Argus evaluates each submitted cell against current validation rules and best result for given sut_timestamp. Marks cell as ERROR if rule finds its value below any rule limit. Validation is skipped if cell status is other than UNSET.
Results to make difference need to change test status depending if are passing or failing. At first idea was to force clients to adjust test status. This commit adds possibility to Argus to track best results for given test and validate incoming results based on rules defined by Argus clients.
List of improvements:
run_id
andresult_date
for future reference. Evaluated only whenhigher_is_better
column metadata is set. Stored in separate table to track the changes (and correctly show validation rules calculated limits for given time)UNSET
.refs: https://github.com/scylladb/qa-tasks/issues/1765