search

MicrosoftPremier / VstsExtensions

Documentation and issue tracking for Microsoft Premier Services Visual Studio Team Services Extensions
MIT License
56 stars 14 forks source link

Cannot use both azurepipelines-coverage.yml file and BQC build task. #127

Closed slee2525FC closed 3 years ago

slee2525FC commented 3 years ago

For each build, I am trying to do both a diff-coverage check and a full-coverage check. I am using the azurepipelines-coverage.yml file for diff-coverage check and then have configured BQC to do my full-coverage check.

I noticed I am no longer getting the comments on my PRs that show the diff-coverage, even though this file has not changed. This has only started happening when I added the BQC build task. Are these two not able to run together?

ReneSchumacher commented 3 years ago

Hi Scott,

I am not aware of any interference between the two features. However, we are currently in the process of disabling the feature flag for coverage data storage to mitigate the timeouts for BQC. I will double-check with the product group to make sure that this disabled feature is not needed for diff-coverage to work.

slee2525FC commented 3 years ago

Ah - understood. Thank you @ReneSchumacher !

Is the disabling of the feature flag a global roll out for all users or just specific users that are using BQC who have reported the time out problem.

I have my full coverage reports using BQC working now and am no longer seeing the timeouts and I don't believe the FF is disabled yet. If this is interfering with the diff-coverage and it's only being disabled on a per-user basis, I'd like to leave this on for us.

ReneSchumacher commented 3 years ago

I am not totally sure about the connection between the feature flag and diff-coverage, that's why I need to discuss with the PG. We only disable the feature flag for customers who reported the timeout issue for BQC, but I was hoping this would be a temporary mitigation.

The root cause - as far as I understand it - is that the testing tools team changed the way coverage data is stored in the backend to speed things up and maybe even allow additional things to work in Azure DevOps. However, the new storage location comes with a problematic delay that prevents BQC from reading coverage data using the REST API in a timely manner. There seems to be another API, which uses the vstmr.dev.azure.com endpoint, which isn't available through our regular client libraries. Thus, BQC would have to dynamically switch from using the library to making a plain REST call, which also only works in the cloud and not for on-premises customers. This and the reason that it seems to be unclear when to use which API is why I'm so reluctant to switch to the new API, esp. since the old API is working for the majority of customers.

I'll keep working with the testing tools team to sort this out. :)

slee2525FC commented 3 years ago

Thanks for the help in trying to solve this for us so far!

ReneSchumacher commented 3 years ago

Hi @slee2525FC,

are you still affected by this issue? I have seen some communication between you and the Azure DevOps product group, but I'm not sure about the outcome.

slee2525FC commented 3 years ago

Hi @ReneSchumacher ,

I'm just getting caught up on emails from over the holiday. I saw the a member of the product group was asking for some additional information, so I'll be working on getting that to them over the next few days. Thanks for following up!

ReneSchumacher commented 3 years ago

Hi @slee2525FC,

can I close this issue? Imho, it is not related to BQC because the task should not interfere with the diff coverage policy that is part of Azure DevOps.

slee2525FC commented 3 years ago

Hi @ReneSchumacher

I'm good with that. I have the open email thread that I can use to continue working with to determine what the specific problem we are seeing is.

Thank you again for all the help!