search

MicrosoftPremier / VstsExtensions

Documentation and issue tracking for Microsoft Premier Services Visual Studio Team Services Extensions
MIT License
57 stars 14 forks source link

Code coverage failed although coverage improvement #11

Closed fplourde closed 6 years ago

fplourde commented 6 years ago

The percentage from our last build is 94.8%, and the current one is 95.3, but the task Check build quality failed.

Note the build had more code but less covered blocks. We are using the attribute [ExcludeFromCodeCoverage] to remove configuration specific code from Code coverage.

Here our configurations: We are on a private TFS VNext server and using Check build quality 5.0.2.

image

Error: The code coverage value (95.339%, 225 blocks) is lower than the value of the previous build! The previous build had a coverage value of 94.8207% (238 blocks).

Logs:

2018-08-01T13:09:48.9276396Z Successfully read code coverage data from build. 2018-08-01T13:09:48.9399880Z Total blocks: 236 2018-08-01T13:09:48.9399995Z Covered blocks: 225 2018-08-01T13:09:48.9400153Z Code Coverage (%): 95.339 2018-08-01T13:09:48.9543273Z Found baseline build with ID 230140. 2018-08-01T13:09:48.9977351Z Successfully read code coverage data from build. 2018-08-01T13:09:49.0127577Z Total blocks: 251 2018-08-01T13:09:49.0127700Z Covered blocks: 238 2018-08-01T13:09:49.0127877Z Code Coverage (%): 94.8207 2018-08-01T13:09:49.0147410Z ##[error]At least one build quality policy was violated. See Build Quality Checks section for more details.

ReneSchumacher commented 6 years ago

Hi Frédéric,

could you check your delta type setting? If it is still set to "absolute" the policy does not compare the percentage values but the absolute blocks covered. As you mentioned yourself, the number of total blocks covered has decreased.

René

fplourde commented 6 years ago

Delta type Absolute Value is indeed checked, but hidden by Force Coverage Improvement .

Funny that a hidden setting impact how the task is running.

Thanks for the help and have a nice day

ReneSchumacher commented 6 years ago

Hi Frédéric,

thanks for the feedback. Of course, you're absolutely right, the hidden option should not affect the policy. I forgot that we force percentage values when using the "Force Coverage Improvement" option. Thus, the issue is indeed a bug. I'll release a new version soon that fixes this problem and automatically compares percentage values in your case.

René

ReneSchumacher commented 6 years ago

Hey,

the issue has been fixed and the new version (v5.0.4) of the task will be released tomorrow. I'm closing the issue now. If you have any additional problems or if the problem occurs again with the new version, please reopen or create a new issue.

Thanks and happy building! René