Incorrect test result view in Azure DevOps after upgrading MSTest to 2.2.4 and higher

mabadev commented 2 years ago

Description

After I updated MSTest.Framework and MSTest.Adapter to 2.2.4 and higher, the test results in Azure DevOps are displayed incorrectly. Each test interation is displayed in a new line. Before that, the interations were shown under a test case.

For example: A Test with 2 iterations

Before: TestCase12

Iteration1
Iteration2

Currently

TestCase12
TestCase12

This a screenshot for the test results on Azure DevOps For example: Test Case 1129

TRX-File look like different After the upgrade, the trx file does not contain an <InnerResults> element

Hier is a screenshot fom trx-file

I use VS 2019 16.11.6 I'm not sure if this is a bug or a new configuration. This is my test configuration: Configure unit tests by using a .runsettings file

Evangelink commented 1 year ago

@dogirala What's the correct repository to track this issue? It doesn't seem to be related to MSTest.

dogirala commented 1 year ago

hello @Evangelink, @charlesdwmorrison. I was OOF last week. Will take a look at this today and get it tracked accordingly.

charlesdwmorrison commented 1 year ago

Any updates on this? My team relies heavily on running tests from test plans.

dogirala commented 1 year ago

@charlesdwmorrison @Evangelink Sorry for the delay in response, I have gone through the issue, and it seems like this is a known issue link that we have created bug a while ago where we observed incorrect associations being happening in visual studio. We have yet to figure out the right team that owns this. I will work upon this to triage this bug to the right team and to assign this discussion to them. Could you help me with the below queries just for my confirmation

Which version of the MS Test Adapter are you using is it 2.2.4+ ?
While doing 'Associate to testcase' in visual studio, did you observe that if you do associate to one testcase in the data driven , all the the data driven rows will get automatically associated to that testcase? Like in the below screenshot

Here I have tried to associate data driven test with input 10 and automatically 20 and 30 are also associated.

charlesdwmorrison commented 1 year ago

I am using version 2.2.10 of the test adapter. (see screenshot below).
Yes, when I use Test Decorations to make data rows above my test method as you show in the above screen shot, then, yes, if I associate one of the datarows with a test case, then the other datarows are also automatically associated to that test case as well. I can see how that might be problematic.

However I did want to mention one additional point: Our data-driven test cases are in Excel. We don't use a simple test Decorations to data-drive our test cases. we use a datasource like the following to connect to an Excel datasheet:

[DataSource("System.Data.OleDb", "Provider=Microsoft.ACE.OLEDB.12.0;" + "Data Source=|DataDirectory|\Data\TestInputData\AccountManagement\GetAccount.xlsx;" + "Persist Security Info=False;Extended Properties='Excel 12.0 Xml;HDR=YES'", "QA_Smoke$", DataAccessMethod.Sequential)]

With our, classic, approach of adding test data to a test method (using an Excel datasheet), there is no opportunity to associate only one of the data rows to a test case ID. The only thing to do is to associate the entire test method to a test case ID. The same would be true, I think, if our test data were kept in SQL Server or Access. The Test Manager does not have an opportunity to inspect data in Excel, SQL Server, or Access, and has no ability to break out these individual rows into distinct test cases.

If there WERE an opportunity to associate only certain rows of Excel data to a test case, I can certainly see how that would be an advantage to a testing effort. However, with the approach of using the Microsoft.ACE.OLEDB test adapter, I don't think there is any opportunity to do this.

The bug, it seems to me, is in how Azure DevOps counts the test case execution. However, I don't know the inner workings of these assemblies.

Here is a screenshot of the MSTest.TestAdapters I am using:

charlesdwmorrison commented 1 year ago

I forgot to mention that the link you posted above (link) seems to go to an MS internal ADO. Do you have any public information on that issue?

Evangelink commented 1 year ago

@dogirala Yeah it'd be good if you had some public ticket tool where community could follow things up. It seems a little weird to have to track AzDo test issues through vstest or testfx repositories.

charlesdwmorrison commented 1 year ago

Update and Question for Microsoft Is the following a supported way to access an Excel data source from a unit test:

[DataSource("System.Data.OleDb", "Provider=Microsoft.ACE.OLEDB.12.0;" + "Data Source=|DataDirectory|\Data\TestInputData\AccountManagement\GetAccount.xlsx;" + "Persist Security Info=False;Extended Properties='Excel 12.0 Xml;HDR=YES'", "QA_Smoke$", DataAccessMethod.Sequential)]

I ask this because:

I can find no official Microsoft references to this approach to accessing an Excel datasource. In official docs, I see the TestContext being advocated, or a config file.
There is at least one blog which hints that paths that are too long interfere with test case counting. See here.
I have been able to get our test solutions to somewhat work from Test Plans with shorter data access paths such as:

[TestMethod] [DataSource("System.Data.Odbc", @"Dsn=Excel Files;dbq=c:\feedback\exceldata\serialnos.xlsx", "Sheet1$", DataAccessMethod.Random)]

As mentioned in my above Nov. 16th post (above), after upgrading to the latest MSTest.Adapter and MSTest.TestFramework, counting test cases is working for us in a build pipeline. But we are still having problems when executing from a test plan.

Is the OleDB approach that I show above deprecated?

Thanks.

Evangelink commented 1 year ago

@charlesdwmorrison I know that's far from ideal answer but I also cannot find much document about it and I know the feature only works for netfx at the moment so I will need some time to investigate the code and past versions to see if there was some change impacting you.

Regarding, AzDo related issues (test plan, test view...) are not under my umbrella so there is nothing much I can do except pinging the related team (which I do).

charlesdwmorrison commented 1 year ago

Is there a link to a github or ms dev forum where I could post the issue of data access to Excel in unit tests more directly?

Evangelink commented 1 year ago

You can try through https://developercommunity.visualstudio.com/. For the GitHub repo, @dogirala please let us know.

dogirala commented 1 year ago

hi @Evangelink, @charlesdwmorrison I am on top of this, please expect an update by tomorrow. From what I checked the issue is not with the testplans rather it is the 'Associate to testcase' functionality of visual studio. I don't have much info regarding the owing team and the public ticket that would reach to them, but I am working on this.

Evangelink commented 1 year ago

If that's Test Explorer side, we have some contacts (cc @AbhitejJohn)

dogirala commented 1 year ago

@AbhitejJohn could you please take a look at the bug and clarify who should be the right team that owns that workflow? I was having a discussion with couple of folks from test plans and there seems to be some confusion around it.

dogirala commented 1 year ago

Had checked with @AbhitejJohn and got the details on the owning team. The bug is assigned Automation Testing team and will be prioritized in the coming sprints. Will follow-up with the team regarding public ticket on the same. Will update on it here. Thanks

Evangelink commented 1 year ago

Closing this ticket as there is nothing left on MSTest side.

charlesdwmorrison commented 1 year ago

Could we get a URL to where the Automation Testing team is tracking this or providing public updates?

Evangelink commented 1 year ago

Hey all,

I have open a ticket on Dev Community https://developercommunity.visualstudio.com/t/Incorrect-test-result-view-in-Azure-DevO/10359804?q=Incorrect+test+result+view for you all to track. Please do upvote the ticket.

dogirala commented 1 year ago

Hello Amaury,

Thanks for the email. Adapting Tcm with Test Adapter latest versions and incorporate any improvements is one of the priority features for us in this semester.

Right now, I am working on going through the older customer tickets we have received regarding the issue customers are facing due to, parity mismatch between the Test Adapter changes and how TCM as a service is consuming the trx file.

Please give me 1-2 days to go over the older tickets and the suggestions you have shared, to come up with a short plan to address the problem and improve the overall Ux for the customers. Below are the key things we are targeting

- Fix all current issues due to data driven changes in latest Test Adapter versions and how trx is file parsed and consumed.
- Check feasibility and Incorporate the suggestions you have shared to improve the overall UI/UX to the customer.
 - Come up with a means to have this parity for future as well, so that if there are any changes done in Test Adapter, corresponding adaptive changes are done in TCM as well.

We would require your support in delivering this, would reach out to the test platform team in 1-2 days to discuss more on this. Appreciate your support.

Thanks and regards, Prabhakar

From: Amaury Levé @.> Sent: Wednesday, May 10, 2023 7:21 PM To: microsoft/testfx @.> Cc: Durga Vijay Prabhakar Ogirala @.>; Mention @.> Subject: Re: [microsoft/testfx] Incorrect test result view in Azure DevOps after upgrading MSTest to 2.2.4 and higher (Issue #1026)

Hey all,

I have open a ticket on Dev Community https://developercommunity.visualstudio.com/t/Incorrect-test-result-view-in-Azure-DevO/10359804?q=Incorrect+test+result+view for you all to track. Please do upvote the ticket.

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/testfx/issues/1026#issuecomment-1542255476, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AYBLNNDV6XYIQ25TG6RPYOTXFOMPDANCNFSM5KN64GTQ. You are receiving this because you were mentioned.Message ID: @.***>

dogirala commented 1 year ago

@charlesdwmorrison Just sharing an update on the issue you mentioned above. We have merged a fix for that issue. It required a two-part fix, one on the visual studio application and the other on the azure devops. The fix on the visual studio would be available from VS 17.8 Preview 3 version. However, the fix would be applicable for all new associations, so the associations need to be done from visual studio.

charlesdwmorrison commented 1 year ago

Thanks. We are familiar with the association of test case process. Not our favorite thing, but we will check it out. Is this the download point for a VS Preview that contains the fix: https://learn.microsoft.com/en-us/visualstudio/releases/2022/release-notes-preview (I see that this says VS 17.8)?

dogirala commented 1 year ago

Hi yes, this is the page. Right now it says the newest is 17.8 preview 2, the next version should have the fix. Also, just to add, the change is under a featureflag TestExecution.ModifiedFqnAssociation, for hosted ado, I am in the process of enabling this shortly. For onprem, please register and enable this FF in config db. Please reach out to us via CSS for any further assistance on the same

b3go commented 8 months ago

Hi yes, this is the page. Right now it says the newest is 17.8 preview 2, the next version should have the fix. Also, just to add, the change is under a featureflag TestExecution.ModifiedFqnAssociation, for hosted ado, I am in the process of enabling this shortly. For onprem, please register and enable this FF in config db. Please reach out to us via CSS for any further assistance on the same

@dogirala using mstest adapter and framework 3.2.0, azure devops server 2022 and self hosted agents with VS 2022 17.8.6. I added this row in tbl_RegistryItems in the tfs_configuration db and restarted the IIS:

PartitionId ParentPath  ChildItem   RegValue
1   #\FeatureAvailability\Entries\TestExecution.ModifiedFqnAssociation\ AvailabilityState\  1

Running tests with a parallel strategy doesn't work. Only the first DataRow is executed, the others are not mentioned (neither .trx, nor Test Result Window:

However running on one agent works, as all DataRows are properly executed and also displayed in Test Resuts:

Unfortunately running on one agent is not possible in this case. These are "unstable" tests that have to rerun if failed (some static cache issues in big legacy app). Which results in this error https://github.com/microsoft/azure-pipelines-tasks/issues/16187 if executed without batch/slicing

Maybe I also missunderstood for what the fix is for :D

dogirala commented 8 months ago

hi @b3go, yes, we've identified this bug and we've fixed this now. Can you share what is the version of Vstest task you are using?. And can you update the Visual studio test task version and give this a try and let me know if you are facing this issue?

b3go commented 8 months ago

Hi @dogirala, had some troubles building the task but I was finally able to get it working. I am now using VSTestV2 2.229.0 but still getting the same issue. With the parallel strategy the slicing of the test cases with DynamicData/DataRow doesn't seem to work. I still get only the first case.

I've found this in the documentation https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/vstest-v2?view=azure-pipelines#i-am-running-into-issues-when-running-data-driven-xunit-and-nunit-tests-with-some-of-the-task-options-are-there-known-limitations As stated in vstest@2 it is not supported to distribute the tests on multiple different agents in xUnit and NUnit. Could it be that this is now also the same with MsTest because since 2.2.4 the test names are generated in the same way as in xUnit and NUnit?

dogirala commented 8 months ago

@b3go apologies for the delay, I see you are using 2.229.0, from what I checked the change we did went into 2.234.0, can you try with any version vstestv2 2.234.0 or more, the distributed scenario should work as expected, please let me know if you are still seeing this issue, I'll try to connect with you to debug it further

b3go commented 8 months ago

@dogirala thanks a lot for your support. I updated the task to 2.235.2 but no luck, only the first case is found and executed:

dogirala commented 7 months ago

@b3go can you help me with couple of queries?

Are these mstest unittests that you are running?
What is the exact scenario in vstest you are running? Is it testassemblies + parallel strategy?

For DynamicData/DataRow, are you overriding the displayName method and setting a customer displayName? --> If yes, can you try generating it something like below?

public string GetDisplayName(MethodInfo methodInfo, object[] data)
  {
      if (data != null)
      {
          var testParamString = new StringBuilder();
          testParamString.Append(methodInfo.Name);
          testParamString.Append(' (');
          var methodParameters = methodInfo.GetParameters();
          for (int i = 0; i < methodParameters.Length; i++)
          {
              testParamString.Append(data[i]);
              testParamString.Append(',');
          }
          testParamString.Length--;
          testParamString.Append(')');

          return testParamString.ToString();
      }
      return methodInfo.Name;
  }

I am trying to understand the exact scenario you are running and the issue you are facing

b3go commented 7 months ago

Hi @dogirala thank you very much for your continued support :)

Are these mstest unittests that you are running?

Yes we currently use MSTest.TestAdapter and TestFramework v3.2.0

What is the exact scenario in vstest you are running? Is it testassemblies + parallel strategy?

This is the relevant pipeline yaml:

- job: run_tests
  strategy:
    parallel: 5
  workspace:
    clean: all
  steps:
    - checkout: none
    - task: DownloadBuildArtifacts@0
      inputs:
        buildType: 'current'
        downloadType: 'single'
        artifactName: 'testdrop'
        itemPattern: '**'
        downloadPath: '$(Build.BinariesDirectory)'
    - task: VSTest@2
      inputs:
        testSelector: 'testAssemblies'
        testAssemblyVer2: 'My.*Test.dll'
        searchFolder: '$(Build.BinariesDirectory)\testdrop'
        testFiltercriteria: 'TestCategory!=Dummy'
        runInParallel: true
        codeCoverageEnabled: false
        distributionBatchType: 'basedOnExecutionTime'
        batchingBasedOnExecutionTimeOption: 'customTimeBatchSize'
        customRunTimePerBatchValue: '480'
        rerunFailedTests: true
        rerunMaxAttempts: '5'

For DynamicData/DataRow, are you overriding the displayName method and setting a customer displayName? --> If yes, can you try generating it something like below?

This is the method signature of that particular test I am testing the behavior with:

[DataTestMethod]
[DataRow(1, BuchungsVorgangArt.Bankbuchung, BewegungsGruppeType.EG, FremdSchadenAbrechnungArtType.Eingang)]
[DataRow(2, BuchungsVorgangArt.Bankbuchung, BewegungsGruppeType.EG, FremdSchadenAbrechnungArtType.Eingang)]
[DataRow(1, BuchungsVorgangArt.Bankbuchung, BewegungsGruppeType.AG, FremdSchadenAbrechnungArtType.Ausgang)]
[DataRow(2, BuchungsVorgangArt.Bankbuchung, BewegungsGruppeType.AG, FremdSchadenAbrechnungArtType.Ausgang)]
[DataRow(3, BuchungsVorgangArt.Umbuchung, default, FremdSchadenAbrechnungArtType.Umbuchung)]
[DataRow(4, BuchungsVorgangArt.Umbuchung, default, FremdSchadenAbrechnungArtType.Umbuchung)]
[DataRow(5, BuchungsVorgangArt.Ausbuchung, default, FremdSchadenAbrechnungArtType.Kosten)]
[DataRow(6, BuchungsVorgangArt.Ausbuchung, default, FremdSchadenAbrechnungArtType.Kosten)]
[DataRow(7, BuchungsVorgangArt.Ausbuchung, default, FremdSchadenAbrechnungArtType.Kosten)]
[DataRow(8, BuchungsVorgangArt.Ausbuchung, default, FremdSchadenAbrechnungArtType.Kosten)]
[DataRow(9, BuchungsVorgangArt.Ausbuchung, default, FremdSchadenAbrechnungArtType.Kosten)]
[DataRow(10, BuchungsVorgangArt.Ausbuchung, default, FremdSchadenAbrechnungArtType.Kosten)]
public void Test_if_CreateFremdschadenAbrechnungStrategy_CreateFremdschadenAbrechnung_works(
    long kontoArtBewegungId,
    BuchungsVorgangArt
    buchungsVorgangArt,
    BewegungsGruppeType bewegungsGruppe,
    FremdSchadenAbrechnungArtType expectedFremdSchadenAbrechnungArt)
{
}

I hope this information helps!

Edit: actually I created a demo repo which results in exactly the same error: https://github.com/b3go/DataRowTests

dogirala commented 7 months ago

@b3go thanks for sharing the details, will check and get back shortly.

b3go commented 6 months ago

@dogirala Hi do you have any updates on this matter?

dogirala commented 6 months ago

Hi @b3go, apologies for the delay, I was out of office, the past couple of weeks, I am back to office this week, I will look into this and create a tracking item for this.

catalin7mar commented 5 months ago

@dogirala Hi! Do you have any updates? At my new company, I discovered that the following issue is still present, which is why we are still using TestAdapter 2.2.3.

Here's the behavior we're observing (Agent Version 2.195.2, TestAdapter 3.4.3):

We run our permission tests on a remote desktop using various test plans from Azure DevOps (Net 6).

When running the tests on my local machine, the result of failed test cases is correct:

However, when I create a release in Azure, the output is not correctly captured:

How it should look (TestAdapter 2.2.3)

dogirala commented 5 months ago

Hi @catalin7mar, are you using ADO onprem server or hosted?, can you share the Organization Name if you are on hosted?

catalin7mar commented 5 months ago

@dogirala Hi! onprem

dogirala commented 5 months ago

@catalin7mar can you please register and turn on this FF --> TestExecution.EnableParentChildForEmptyDataDrivenDynamicTests

For registration, you can follow the below steps -

Execute the below on configuration DB, replace the {featureflagname} with the above highlighted

INSERT INTO tbl_RegistryItems (PartitionId, ParentPath, ChildItem, RegValue)

VALUES (1,'#\FeatureAvailability\Definitions{FeatureFlagName}\', 'Description\', 'Flag for controlling enabling of association to testcase workflows with modified Fully Qualified Name'), (1,'#\FeatureAvailability\Definitions{FeatureFlagName}\', 'Name\', '{FeatureFlagName}'), (1,'#\FeatureAvailability\Definitions{FeatureFlagName}\', 'Owner\', '{Service}')

EXEC prc_SetRegistryValue 1, '#\FeatureAvailability\Entries{FeatureFlagName}\AvailabilityState\', 1

Restart the IIS Web App for the Azure DevOps collection.

b3go commented 3 months ago

Hi @dogirala me again :D

how does TestExecution.EnableParentChildForEmptyDataDrivenDynamicTests add to TestExecution.ModifiedFqnAssociation? (btw I think you missed a \ between Defintions/Entries and {FeatureFlagName})

Could be that I actually made a mistake enabling the Feature Flag, I only added the row manually

PartitionId ParentPath  ChildItem   RegValue
1   #\FeatureAvailability\Entries\TestExecution.ModifiedFqnAssociation\ AvailabilityState\  1

So what I would do now is:

INSERT INTO tbl_RegistryItems (PartitionId, ParentPath, ChildItem, RegValue)

VALUES
(1,'#\FeatureAvailability\Definitions\TestExecution.ModifiedFqnAssociation', 'Description', 'Flag for controlling enabling of association to testcase workflows with modified Fully Qualified Name'),
(1,'#\FeatureAvailability\Definitions\TestExecution.ModifiedFqnAssociation', 'Name', 'TestExecution.ModifiedFqnAssociation'),
(1,'#\FeatureAvailability\Definitions\TestExecution.ModifiedFqnAssociation', 'Owner', '{Service}')

EXEC prc_SetRegistryValue 1, '#\FeatureAvailability\Entries\TestExecution.ModifiedFqnAssociation\AvailabilityState', 1

What would be the value for {service}?

Thanks!

microsoft / testfx

Incorrect test result view in Azure DevOps after upgrading MSTest to 2.2.4 and higher #1026

Description