Closed aapalm closed 6 years ago
Hi Aaron,
thank you for pointing out this issue. We currently have a fixed precision when calculating percentages for code coverage. I'll update the policy to use the precision defined by the variance. The update should go out tomorrow.
René
Hi Aaron,
I have updated the code coverage policy and increased the standard precision from two decimals to four decimals. This picks up a change of one element in a million elements (e.g., one block in a million). Setting a smaller variance will automatically increase the precision (e.g., a variance of 0.00001 will pick up a change of one block in ten million blocks). The new version 5.0.0 will be released today.
I'm closing this issue now. If it still doesn't work for you or if you need any additional assistance, please let us know.
René
Based on our observations, it seems the Variance for percentage deltas can only have a precision of 2 decimals, which does not allow for small variance (when denominator is large).
For example, we use our previous builds to compare block coverage using percentage delta, and allow for a "0.006" variance (appears to be percentage based when the delta is percentage), to allow for around 30 blocks to pass. This should allow for the following case to pass:
But, as the screenshot below shows, this actually failed:
Can you let us know what we need to do to allow for 30 blocks to pass for percentage deltas, given large denominators (like the above case)?