##[warning]Test Impact (Run only Impacted tests) and Code Coverage data collector will not work.

[Bouke]

Question / Bug

Enter Task Name: VSTest

Environment

• Server - Azure Pipelines or TFS on-premises?

  https://dev.azure.com/cloudbilling/CloudBilling/_build/results?buildId=18617&view=logs&j=bf94e6da-eeef-50d1-100b-0ed7f4d964f2&t=2c85ca81-ba16-5100-d932-92222da60a57

• Agent - Hosted or Private:

Agent name: 'Azure Pipelines 5'
Agent machine name: 'fv-az78-882'
Current agent version: '2.193.1'
Operating System
Microsoft Windows Server 2019
10.0.17763
Datacenter
Virtual Environment
Environment: windows-2019
Version: 20211018.0

Issue Description

Since today my VSTest task is printing this warning:

[warning]Test Impact (Run only Impacted tests) and Code Coverage data collector will not work.

There's no code change on my end, nor have I changed the build pipeline. I have cross-verfied against a prior commit for which this warning was not printed: running the pipeline against that commit now yields that warning.

I have no idea what this warning means. I know I do want code coverage.

[andyfisher100]

We have seen the same issue on our pipelines. For what its worth our build agent is using VS2019 and the pipeline is using the "Visual Studio test platform installer" task which was pulling the latest (v17) test tools that are for the upcoming release of VS2022

[tomaustin700]

Yep we're seeing this also, I have exactly the same setup as @andyfisher100

[andyfisher100]

Just for clarification. We set the Visual Studio test platform installer task to use v16.11.0 of the test tools and the issue went away

[Bouke]

Downgrading 17 to 16 resolved this problem for me as well. For reference the relevant build step:

    - task: VisualStudioTestPlatformInstaller@1
      inputs:
          packageFeedSelector: 'nugetOrg'
          versionSelector: 'specificVersion'
          testPlatformVersion: '16.11.0'

[jfbeaulac]

Seems to be caused by this modification in the Visual Studio Test Platform: https://github.com/microsoft/vstest/pull/3068

The VSTest Task outputs this warning if it cannot find "Microsoft.IntelliTrace.ProfilerProxy.dll" in the VSTest package.

[Jaans]

Does this mean that the new version of VS and DevOps will no longer support code coverage with test runs?

[tomaustin700]

Does this mean that the new version of VS and DevOps will no longer support code coverage with test runs?

I'd assume it all works fine if the agent is running VS2022 and the latest test tools.

[dmytro-gokun]

@tomaustin700 Nope. Still the same error when running VS2022 and the latest test tools ...

[jakubch1]

On VS 2022 and TestPlatform 17.0 Intellitrace is no longer needed to generate code coverage so maybe this warning should be removed/ignored. Could you please confirm that code coverage is really not generated? I saw 1 pipeline where code coverage was generated and this warning was in logs.

[tbolon]

I also encountered this problem today.

We wanted to migrate from VS2019 vstest.console.exe because it is still compiled against .NET 4.5.1 and still use TLS 1.0 when running tests, but now that we use v17 of vstest.console.exe we now receive this warning.

Perhaps we could suggest a PR to remove this warning? Do you know if there is a way to get rid of this warning when invoking vstest.console.exe? Which code is responsible for writing this warning? I could not find any trace of this text in vstest repo or this one. Maybe it's some trace adapter ?

[tbolon]

Ok I just found the resource containing this warning. Maybe it's just this task which should be tweaked.

[tbolon]

And the function which emits a warning if the dll Microsoft.IntelliTrace.ProfilerProxy.dll is not found.

Because @jakubch1 said that the IntelliTrace is not expected to be found when using v17, how this function could be updated to handle this case?

There is also a reference to this resource in helpers.ts.

[andy-mishechkin]

Hello! I've faced up with the same issue in "Visual Studio test platform installer" task. Our smore tests are not worked correctly, when I select "Latest Stable" in Version field of this task, and I see this warning in deployment log. But when I select "Specific version" and "Test Platform Version" 16.11.0 - all works fine. May you fix it in the last stable version

[kaylumah]

Ran into this issue after updating VisualStudioTestPlatformInstaller has any progress been made regarding this?

[ashish-tamhane]

Running into the same issue. Any progress on this yet?

[jakubch1]

@andy-mishechkin @kaylumah @ashish-tamhane is code coverage really missing or you are just getting this warning?

[ashish-tamhane]

@andy-mishechkin @kaylumah @ashish-tamhane is code coverage really missing or you are just getting this warning?

I am just getting this warning. Code Coverage is being generated perfectly.

[jakubch1]

@phanikmmsft could you please simply remove this warning from your code? Code coverage is using now CLR Instrumentation Engine and Intellitrace files are not needed anymore so this warning is useless.

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[krokofant]

Still a problem

[dougclutter]

Still a problem in 17.4.1...is anybody going to fix this?

[tbolon]

It's really problematic, because my warning are filled with this line, and I don't see the real warnings easily on my build reports.

@jakubch1 could you please update the assignee for this issue? it seems @phanikmmsft does not work on this project anymore?

[kimballry]

Still occurring with 17.5.0-preview

[jakubch1]

@abhadauria1 any update on removing this warning?

[MiYanni]

This issue is still occurring. Bumping thread.

[filetrail-dgrossman]

We are experiencing this superfluous warning, as well.

[mvonballmo]

I'm seeing this warning three times in my output. I have a single test task with a single test. It warns three times. This issue has been open for two years. Is it really possible that there's no workaround and no fix for the latest version of the standard testing task?

[akytsun]

I have the same issue in 17.6.3 , when that will be fixed???

[mvonballmo]

For what it's worth, my workaround is to roll back to 16.11.0, which is the last release of the VS2019 test runner. It seems every version of the VS2022 test runner has the warnings.

Include the following in the pipeline (relevant items only; not a complete pipeline definition).

- task: VisualStudioTestPlatformInstaller@1
  inputs:
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'specificVersion'
    testPlatformVersion: '16.11.0'

- task: VSTest@2
  inputs:
    vsTestVersion: 'toolsInstaller'

Edit: I just realized upon re-scanning the comments that this solution had already been offered a while ago. 🤷‍♂️

[ms6073]

This issue is still occurring 22-months after the initial report:

2023-08-14T17:22:53.1189389Z Task : Visual Studio Test 2023-08-14T17:22:53.1189480Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2). 2023-08-14T17:22:53.1189930Z Version : 2.224.0 2023-08-14T17:22:53.1189983Z Author : Microsoft Corporation 2023-08-14T17:22:53.1190063Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/vstest 2023-08-14T17:22:53.1190330Z ============================================================================== 2023-08-14T17:22:56.0000189Z Running tests using vstest.console.exe runner. 2023-08-14T17:22:56.0000747Z ====================================================== 2023-08-14T17:22:56.0001762Z Test selector : Test assemblies 2023-08-14T17:22:56.0002114Z Test filter criteria : TestCategory=Adhoc 2023-08-14T17:22:56.0002435Z Search folder : D:\a\1\s 2023-08-14T17:22:56.0237296Z Action when minimum tests threshold not met : donothing 2023-08-14T17:22:56.0237491Z Minimum tests expected to be run: 0 2023-08-14T17:22:56.2921144Z Run settings file : D:\a\1\s\QAAutomation\UITests\UiTests\bin\Release\net7.0\tst.edge.runsettings 2023-08-14T17:22:56.2922294Z Run in parallel : false 2023-08-14T17:22:56.2922609Z Run in isolation : false 2023-08-14T17:22:56.2922930Z Path to custom adapters : undefined 2023-08-14T17:22:56.2924324Z Other console options : /logger:trx;LogFileName=tst-edge-adhoc.trx /framework:".NETCoreApp,Version=v7.0" 2023-08-14T17:22:56.2924736Z Code coverage enabled : false 2023-08-14T17:22:56.2925299Z Diagnostics enabled : false 2023-08-14T17:22:56.2929939Z Run the tests locally using vstest.console.exe 2023-08-14T17:22:56.2930121Z ======================================================== 2023-08-14T17:22:56.2930747Z Source filter: D:\a\1\s\QAAutomation\UITests\UITests\bin\Release\net7.0*tests.dll 2023-08-14T17:22:56.6963233Z ##[warning]Test Impact (Run only Impacted tests) and Code Coverage data collector will not work. 2023-08-14T17:22:56.7306337Z ##[warning]Test Impact (Run only Impacted tests) and Code Coverage data collector will not work. 2023-08-14T17:22:56.8691061Z ##[warning]Test Impact (Run only Impacted tests) and Code Coverage data collector will not work.

[ms6073]

For what it's worth, my workaround is to roll back to 16.11.0, which is the last release of the VS2019 test runner. It seems every version of the VS2022 test runner has the warnings.

Include the following in the pipeline (relevant items only; not a complete pipeline definition).

- task: VisualStudioTestPlatformInstaller@1
  inputs:
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'specificVersion'
    testPlatformVersion: '16.11.0'

- task: VSTest@2
  inputs:
    vsTestVersion: 'toolsInstaller'

While the comments above offer a valid workaround, one has to ask how is it that there have been so many newer versions published (latest stable version is 17.6.3) yet this issue has never been addressed?

[xinfli]

I just tried the latest stable version 17.6.3, the warning is still there, now our pipeline raised the warning more than 300 times, other problems are hidden in the warning flood...

[kic001]

Same issue here. All releases after 16.11 will result i a flood of warnings. Tryed out new xunit 2.5 and newest vstest runner.

[jesperll]

Dear Microsoft, please fix this annoying bug

[quasarea]

@jakubch1 could you please update the assignee for this issue? it seems @abhadauria1 does not work on this project anymore? Maybe @Haplois could take a look at this as he created this issue with his change https://github.com/microsoft/vstest/pull/3068

[adityashahms]

With VsTest task version 2.235.0, this issue should be resolved while consuming the latest test platform versions ( > 17.0.0).

[dougclutter]

@adityashahms

With VsTest task version 2.235.0, this issue should be resolved while consuming the latest test platform versions ( > 17.0.0).

That's great news! I can finally stop explaining that message to my managers. :)

We're using the VSTest@2 task, but it's still using 2.229.0. When will 2.235.0 be pushed broadly to Azure DevOps? Is there something we need to do?

[adityashahms]

The changes were merged yesterday only. It would be gradually rolled out to all the Azure DevOps organization within next 15-20 days. No changes are required from your end. As and when the deployment happens for your organization, latest task version would be picked up. I'll update the thread if things go south.

[adityashahms]

The latest VsTest Task version (>=2.235.0) is working as expected and should be available to consume where logging of unnecessary warnings isn't happening. Hence, closing the issue. Please feel free to reopen the issue if it persists.

[quasarea]

Hi @adityashahms, was this change deployed to vstest@3 as well? I'm currently on

Starting: VSTest
==============================================================================
Task         : Visual Studio Test
Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
Version      : 3.231.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/vstest
==============================================================================

and still getting this error

##[warning]Test Impact (Run only Impacted tests) and Code Coverage data collector will not work.

Not sure if I should downgrade to vstest@2 as changes are deployed there or what's going on with those task versions.

[adityashahms]

@quasarea , please downgrade to VsTest@2, since VsTest@3 isn't getting active updates as much as VsTest@2.

[ms6073]

FYI - At the top of the page I see that this issue is marked as closed, so unsure if anyone is really paying attention. Might be a good idea to open a new case for the VsTest@3.