search

iterative / dvc

🦉 ML Experiments and Data Management with Git
https://dvc.org
Apache License 2.0
13.36k stars 1.16k forks source link

metrics diff atol and rtol #10408

Open wiktorowski211 opened 2 months ago

wiktorowski211 commented 2 months ago

It would be useful to be able to filter the results of the dvc metrics diff command based on absolute or relative tolerance.

This type of solution allows, first of all, to filter negligible changes but also, for example, to build jobs on CI that fail if any of the metrics have changed too drastically.

Currently, this can be done with a custom python script, or using AWK:

dvc metrics diff | awk '($5 > ${metrics_absolute_tolerance} || $5 < -${metrics_absolute_tolerance}) {print; should_fail=1} END {exit should_fail}''

Looking forward for your feedback!

### Tasks