Build Quality Checks: usage with pull requests from GitHub instead of Azure Repos

frankieyan commented 4 years ago

Hi there! I'm trying to set up Build Quality Checks with our project using Azure Pipelines + GitHub, but I'm a little lost when setting up the base branch to compare code coverage against.

For the two Build Service users, we only seem to see permissions around builds, ie. "Update build information", rather than git related permissions like "Contribute to pull requests". Is this due to not having Azure Repos? If so, is it possible for this to work with a project on Github?

Thanks in advance 😃

frankieyan commented 4 years ago

Attaching the configs just in case there's anything obvious missing:

    - task: PublishCodeCoverageResults@1
      inputs:
          codeCoverageTool: 'cobertura'
          summaryFileLocation: $(System.DefaultWorkingDirectory)/coverage/cobertura-coverage.xml
      displayName: 'Publish code coverage result'

    - task: BuildQualityChecks@7
      inputs:
          checkWarnings: true
          checkCoverage: true
          coverageType: lines
          baseBranchRef: $(System.PullRequest.TargetBranch)
      displayName: 'Check build quality'

ReneSchumacher commented 4 years ago

Hi Frankie,

the task also works with GitHub repos but it is not able to post any policy status or pull request comments to the GitHub pull request. This is only available with Azure Repos. However, it should still be able to break your build based on coverage values or warnings.

What exactly do you want to check with the policy? Or what isn't working to your expectations?

René

frankieyan commented 4 years ago

Hi René,

I'm trying to fail the build when a pull request lowers the coverage compared the target branch. Even though I've specified $(System.PullRequest.TargetBranch) in my baseBranchRef, I was still seeing baseline build ID's for previous builds of the same branch. I've also tried setting the branch name directly with no luck. Since there were errors about insufficient permissions I assumed that was the issue (please don't laugh at the % 😛 I'm only running a few tests to speed up the trial and error process):

Validating code coverage policy...
There was an error reading pull request statuses. Please ensure that the build account has sufficient permissions (see https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/PullRequests.md#prerequisites-1)!
Waiting for code coverage data...
Waiting for code coverage data...
Waiting for code coverage data...
Successfully read code coverage data from build.
Evaluating coverage data from 1 filtered code coverage data sets...
Total lines: 32569
Covered lines: 1299
Code Coverage (%): 3.9885
Found baseline build with ID 47396.
Successfully read code coverage data from build.
Evaluating coverage data from 1 filtered code coverage data sets...
Total lines: 32569
Covered lines: 1299
Code Coverage (%): 3.9885
[SUCCESS] Code coverage policy passed with 3.9885% (1299/32569 lines).
There was an error reading pull request statuses. Please ensure that the build account has sufficient permissions (see https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/PullRequests.md#prerequisites-1)!
Finishing: Check build quality

If that isn't it, could you point me to where I might have misconfigured something? Thanks!

ReneSchumacher commented 4 years ago

Hi again,

I realized that this is a bug in the task. It's not working properly because the pull request variables do not contain the full branch ref when the build is triggered by GitHub. I'm fixing the issue right now so there should be an updated version soon.

René

ReneSchumacher commented 4 years ago

HI @frankieyan,

as promised this issue is now fixed with v7.2.1 of the task that just went live. You should now be able to use the variable $(System.PullRequest.TargetBranch) as described in the docs.

Happy building, René

frankieyan commented 4 years ago

Thanks René - I will test it out and report back 👍

frankieyan commented 4 years ago

Hello again @ReneSchumacher

I've updated to v7.2.1 but I still couldn't get the baseline branch to be recognized.

Here are the logs from the base branch. This is run against our main branch which doesn't contain any coverage data yet:

Starting: Check build quality
==============================================================================
Task         : Build Quality Checks
Description  : Breaks a build based on quality metrics like number of warnings or code coverage.
Version      : 7.2.1
Author       : Microsoft Premier Services
Help         : [[Docs]](https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/overview.md)
==============================================================================
SystemVssConnection exists true
Using IdentifierJobResolver
Validating build warnings policy...
Running task warnings analysis...
[WARNING] The specified task filters did not match any build task.
##[warning]The specified task filters did not match any build task.
Overall total warnings: 0
Overall filtered warnings: 0
No baseline build for current branch found. Looking for builds on pull request target branch.
[WARNING] The baseline build could not be found. All policies based on the previous build will pass.
##[warning]The baseline build could not be found. All policies based on the previous build will pass.
[SUCCESS] Warnings policy passed with 0 warning(s).
Validating code coverage policy...
There was an error reading pull request statuses. Please ensure that the build account has sufficient permissions (see https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/PullRequests.md#prerequisites-1)!
Waiting for code coverage data...
Waiting for code coverage data...
Waiting for code coverage data...
Successfully read code coverage data from build.
Evaluating coverage data from 1 filtered code coverage data sets...
Total lines: 32614
Covered lines: 2391
Code Coverage (%): 7.3312
No baseline build for current branch found. Looking for builds on pull request target branch.
[SUCCESS] Code coverage policy passed with 7.3312% (2391/32614 lines).
There was an error reading pull request statuses. Please ensure that the build account has sufficient permissions (see https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/PullRequests.md#prerequisites-1)!
Finishing: Check build quality

Here's the output from the follow up branch. When I checked the variables in the run options I see the correct branch names being set:

Starting: Check build quality
==============================================================================
Task         : Build Quality Checks
Description  : Breaks a build based on quality metrics like number of warnings or code coverage.
Version      : 7.2.1
Author       : Microsoft Premier Services
Help         : [[Docs]](https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/overview.md)
==============================================================================
SystemVssConnection exists true
Using IdentifierJobResolver
Validating build warnings policy...
Running task warnings analysis...
[WARNING] The specified task filters did not match any build task.
##[warning]The specified task filters did not match any build task.
Overall total warnings: 0
Overall filtered warnings: 0
No baseline build for current branch found. Looking for builds on pull request target branch.
[WARNING] The baseline build could not be found. All policies based on the previous build will pass.
##[warning]The baseline build could not be found. All policies based on the previous build will pass.
[SUCCESS] Warnings policy passed with 0 warning(s).
Validating code coverage policy...
There was an error reading pull request statuses. Please ensure that the build account has sufficient permissions (see https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/PullRequests.md#prerequisites-1)!
Waiting for code coverage data...
Waiting for code coverage data...
Waiting for code coverage data...
Successfully read code coverage data from build.
Evaluating coverage data from 1 filtered code coverage data sets...
Total lines: 32569
Covered lines: 1299
Code Coverage (%): 3.9885
No baseline build for current branch found. Looking for builds on pull request target branch.
[SUCCESS] Code coverage policy passed with 3.9885% (1299/32569 lines).
There was an error reading pull request statuses. Please ensure that the build account has sufficient permissions (see https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/PullRequests.md#prerequisites-1)!
Finishing: Check build quality

I did notice that with both builds, the message No baseline build for current branch found. Looking for builds on pull request target branch now appears which wasn't there before. Could it still have something to do with the pull request status permissions?

ReneSchumacher commented 4 years ago

Hi again,

I'm very sorry that the issue still persists. I'm sure it worked in my setup so let's check that I fully understood your environment so I can set up a similar one on my side:

Your code is on GitHub, right?
You have a pull request for merging from one complex branch (frankie/test-coverage-change) to another complex branch (frankie/code-coverage-reporting). - So far, I only tested from simple branch (e.g., ccTest) to master.
You have a yaml pipeline definition in your repo that is executed in Azure Pipelines and has a pull request trigger definition, right?
The configuration for the Build Quality Checks task still matches the yaml snippet you posted above?

Could you please add the variable system.debug: true to your pipeline definition, run the builds again, and send over the logs for the Build Quality Checks task to PSGerExtSupport@microsoft.com? I'll take a closer look at what's going on and try to set up the same environment to figure out why it still isn't working as expected.

René

ReneSchumacher commented 4 years ago

Hi @frankieyan,

I finally found the issue and solved it. We're running a couple last tests and will then publish the new version (v7.2.2) to the marketplace. We also removed the unnecessary warning about the permissions sine that is just confusing if you're not running your pull request in Azure Repos.

Happy building! René

frankieyan commented 4 years ago

Hi René,

Unfortunately, I still couldn't get it to find the baseline build on v7.2.2. I've sent the logs over to the email address you provided. Thanks again and I really appreciate all the efforts you've put in 😊

frankieyan commented 4 years ago

Just a follow up after chatting with René in case anyone runs into something similar - this was my mistake as my base branch was running its build with a merge PR in the form of refs/pull/xxxx/merge rather than on the branch itself, which is why my follow up PR's build wasn't able to find the baseline when provided with its branch name. Once I initiated a build on the baseline branch on its own, the follow-up build was able to find it correctly 👍

Magnuti commented 1 year ago

@frankieyan are you able to provide example code for the new baseBranchRef for the branch.

And one more thing, is it possible to compare the PR branch to master/main? If so, how?

frankieyan commented 1 year ago

Sorry @Magnuti, we've moved off of Azure for a while now so I don't have any context on this anymore

Magnuti commented 1 year ago

No worries @frankieyan, maybe the authors knows this? @ReneSchumacher

MicrosoftPremier / VstsExtensions

Build Quality Checks: usage with pull requests from GitHub instead of Azure Repos #102