Quality check fails if code coverage values are exact

[Dismissile]

If I have a unit test project that has 80% coverage over 5,000 blocks of code and I make a configuration change and run a build, my build will fail because code coverage remained the exact same value (no code was updated).

I feel this should be configurable so that values that are the exact same do not trigger errors.

[ReneSchumacher]

Hey Travis,

I just replied to your question on the Visual Studio Marketplace so I just copy my answer over here so that others reading the issue have the same information:

If you enable the Force Coverage Improvement option, there is no way to prevent the task from failing the build unless your coverage value increased. As you mentioned you can specify an upper threshold above which the task will stop failing. If you don't want to do this, you currently only have the option to disable the Force Coverage Improvement option, which ensures that coverage does not decrease (i.e., it has the exact same or higher value) without forcing every build to have higher code coverage.

You might be looking for a way to force improvement but only if there have been changes to a file that is relevant for code coverage. However, it is not that easy to filter out irrelevant files as we want to make sure that the task can be used for any technology and any type of coverage. Thus, we'd have to add another configuration option to filter out specific file types that users would have to specify. Is that what you're looking for?

René

[Dismissile]

Couldn’t there also be a checkbox along the lines of:

“Pass coverage checks if coverage values remain the same”

That way it fails if coverage has decreased, but not if it hasn’t increased. I’m also curious what would happen if I ran the same build on the same commit twice in a row. I assume there would be no difference in coverage values between the two and therefore it would fail?

Travis Ellis Solution Consultant Mobile: 402.415.6398 Deliveron Consulting Services [cid:image009.png@01D1E822.C4FFDE10]http://www.deliveron.com/ [cid:image016.png@01CF0620.0729E520]https://www.facebook.com/deliveron [cid:image017.png@01CF0620.0729E520]https://twitter.com/deliveron [cid:image018.png@01CF0620.0729E520]http://www.linkedin.com/company/deliveron-consulting-services

From: René Schumacher notifications@github.com Sent: Tuesday, July 23, 2019 12:44 PM To: MicrosoftPremier/VstsExtensions VstsExtensions@noreply.github.com Cc: Travis Ellis travisellis@deliveron.com; Author author@noreply.github.com Subject: Re: [MicrosoftPremier/VstsExtensions] Quality check fails if code coverage values are exact (#41)

Hey Travis,

I just replied to your question on the Visual Studio Marketplace so I just copy my answer over here so that others reading the issue have the same information:

If you enable the Force Coverage Improvement option, there is no way to prevent the task from failing the build unless your coverage value increased. As you mentioned you can specify an upper threshold above which the task will stop failing. If you don't want to do this, you currently only have the option to disable the Force Coverage Improvement option, which ensures that coverage does not decrease (i.e., it has the exact same or higher value) without forcing every build to have higher code coverage.

You might be looking for a way to force improvement but only if there have been changes to a file that is relevant for code coverage. However, it is not that easy to filter out irrelevant files as we want to make sure that the task can be used for any technology and any type of coverage. Thus, we'd have to add another configuration option to filter out specific file types that users would have to specify. Is that what you're looking for?

René

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftPremier/VstsExtensions/issues/41?email_source=notifications&email_token=ACPYJJRJO4IOM656RBIZNQLQA47PVA5CNFSM4IGG5P62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2T4VXI#issuecomment-514312925, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ACPYJJV6LCDFW6NYRJICYSTQA47PVANCNFSM4IGG5P6Q.

[ReneSchumacher]

That option already exists. Just disable the Force Coverage Improvement option. It will not fail the build unless the coverage value has decreased.

[Dismissile]

Oh I assumed it would only fail builds unless it met a threshold percentage in that case. Thanks for the clarification.

Travis Ellis Solution Consultant Mobile: 402.415.6398 Deliveron Consulting Services [cid:image009.png@01D1E822.C4FFDE10]http://www.deliveron.com/ [cid:image016.png@01CF0620.0729E520]https://www.facebook.com/deliveron [cid:image017.png@01CF0620.0729E520]https://twitter.com/deliveron [cid:image018.png@01CF0620.0729E520]http://www.linkedin.com/company/deliveron-consulting-services

From: René Schumacher notifications@github.com Sent: Tuesday, July 23, 2019 12:54 PM To: MicrosoftPremier/VstsExtensions VstsExtensions@noreply.github.com Cc: Travis Ellis travisellis@deliveron.com; Author author@noreply.github.com Subject: Re: [MicrosoftPremier/VstsExtensions] Quality check fails if code coverage values are exact (#41)

That option already exists. Just disable the Force Coverage Improvement option. It will not fail the build unless the coverage value has decreased.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftPremier/VstsExtensions/issues/41?email_source=notifications&email_token=ACPYJJRMWRFQFT32IDVAAUDQA5ASPA5CNFSM4IGG5P62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2T5PWI#issuecomment-514316249, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ACPYJJXXNDYM7OC3EGIKU3DQA5ASPANCNFSM4IGG5P6Q.

[ReneSchumacher]

To fail a build based on a simple threshold, you'd have to set Fail Build On to Fixed Threshold. Whenever you set it to Previous Build, the policy automatically validates that the current coverage value doesn't fall below the previous value. That is the default behavior.

There are teams that want to drive up code coverage values, though. Those teams can leverage the force option, as it forces them to add more tests and cover more code with every build. And since you rarely reach 100% code coverage (and I'd never recommend to even try to bring it up that high unless you're doing TDD and have no generated code), you can specify an additional upper threshold with the force option so that the policy still passes when coverage does not increase as long as it is at or above the upper threshold.

Hope that makes all the options a bit clearer. If you need anything else, just let me know.

Happy building! René