search

softwarelanguageslab / maf

Static Analysis Framework for Modular Analyses
Other
13 stars 12 forks source link

Automate precision tests and track precision across commits #15

Open acieroid opened 3 years ago

acieroid commented 3 years ago

Having a Github Action to automate comparing performance/precision between different commits would be useful to track regressions and improvements.

Doing it on a commit-per-commit basis is useful for precision, but can be problematic for performance : having a +1% hit is not a problem, but having a +1% hit on each of 100 commits is problematic.

bramvdbogaerde commented 2 years ago

Isn't this already (partially) solved by our current Github Actions and the corresponding graphs we generate?

Or did you have a more complete alerting system in mind?

acieroid commented 2 years ago

For performance, this is indeed solved (modulo some stability considerations). For precision though, we are currently generating text reports but not tracking the evolution of precision. Hence, so far we probably never look at these precision numbers. Automating the comparison process (i.e., having an action alert us when precision decreases or increases), similar to the performance action, would definitely help I think.

I updated the title of this issue to reflect the current status.