There comes a time, it seems, in any team, when you have to port a codebase, or update a major framework, or dramatically change some tooling and, inevitably, a ton of unit tests fail, for some period.
Someone usually decides to work up some stats on the data from, say, the last run of the tests. They might present a table like this...
And maybe they'd present this information day-over-day or week-over-week until the number are "good" or the effort is over.
I'm having a really hard time with these numbers...
What does it mean to be 89% successful?
What does it mean if all your failures are caused by a single problem, that can be fixed once?
How are you using continuous terms to describe discrete data (pass/fail)?
How does one project relate to another? How does one test relate to another (in the same project, even)?
Problems...
Continuous vs. discrete data and methods
Data may not be related over time
"Average" is only appropriate for normal distributions
Considerations...
A unit tests should have a single assertion, so that the result (pass/fail) is 1:1 with the assertion (if you have multiple assertions, a test might only be 75% failing -- WAT)
A test is either pass/fail, a test suite (project?) is either pass/fail, a test run is either pass/fail. The concept of partially passing or failing needs to be nonsense.
There comes a time, it seems, in any team, when you have to port a codebase, or update a major framework, or dramatically change some tooling and, inevitably, a ton of unit tests fail, for some period.
Someone usually decides to work up some stats on the data from, say, the last run of the tests. They might present a table like this...
And then they would conclude...
And maybe they'd present this information day-over-day or week-over-week until the number are "good" or the effort is over.
I'm having a really hard time with these numbers...
Problems...
Considerations...
Todo...