Percentage based coverage does not allow for small variance due to round-off?

[aapalm]

Based on our observations, it seems the Variance for percentage deltas can only have a precision of 2 decimals, which does not allow for small variance (when denominator is large).

For example, we use our previous builds to compare block coverage using percentage delta, and allow for a "0.006" variance (appears to be percentage based when the delta is percentage), to allow for around 30 blocks to pass. This should allow for the following case to pass:

new coverage: 83734 / 347662 = 0.24084887045463697499295292554262

baseline coverage: 83734 / 347660 = 0.24085025599723868147040211701087

difference: 0.006 > 0.0000013855426017064774491914682526917 (PASS)

But, as the screenshot below shows, this actually failed:

Can you let us know what we need to do to allow for 30 blocks to pass for percentage deltas, given large denominators (like the above case)?

[almtcger]

Hi Aaron,

thank you for pointing out this issue. We currently have a fixed precision when calculating percentages for code coverage. I'll update the policy to use the precision defined by the variance. The update should go out tomorrow.

René

[ReneSchumacher]

Hi Aaron,

I have updated the code coverage policy and increased the standard precision from two decimals to four decimals. This picks up a change of one element in a million elements (e.g., one block in a million). Setting a smaller variance will automatically increase the precision (e.g., a variance of 0.00001 will pick up a change of one block in ten million blocks). The new version 5.0.0 will be released today.

I'm closing this issue now. If it still doesn't work for you or if you need any additional assistance, please let us know.

René