search

microsoft / testfx

MSTest framework and adapter
MIT License
714 stars 253 forks source link

Extra test cases for DataRow methods #488

Closed wujysh closed 2 years ago

wujysh commented 6 years ago

Description

After upgrading MSTest to MSTest v2, there is an extra test case for each DataRow method (like a summary to all data row tests of the method). And that causes the number of total tests, passed and failed are also not expected. It is fine with MSTest v1 and vstest.cosole.exe.

Is this a known issue or expected feature? Is there any parameters can be set to change the behavior?

Steps to reproduce

    [DataTestMethod]
    [DataRow("1")]
    [DataRow("2")]
    public void TestMethodA(string a)
    {
        Assert.IsTrue(true);
    }

    [DataTestMethod]
    [DataRow("1")]
    [DataRow("2")]
    public void TestMethodB(string a)
    {
        Assert.IsTrue(a.Equals("1"));
    }

Expected behavior

Passed   TestMethodA (Data Row 0)
Passed   TestMethodA (Data Row 1)
Passed   TestMethodB (Data Row 0)
Failed   TestMethodB (Data Row 1)
Total tests: 4. Passed: 3. Failed: 1. Skipped: 0.

Actual behavior

Passed   TestMethodA
Passed   TestMethodA (Data Row 0)
Passed   TestMethodA (Data Row 1)
Failed   TestMethodB
Passed   TestMethodB (Data Row 0)
Failed   TestMethodB (Data Row 1)
Total tests: 6. Passed: 4. Failed: 2. Skipped: 0.

Environment

Run tests through vstest.console.exe on Azure DevOps MSTest 1.3.2

dfrancoed commented 6 years ago

I'm getting the same behavior, and I trying to figure out if it's an issue maybe with the default console logger. https://github.com/Microsoft/testfx/issues/484

cltshivash commented 5 years ago

@dfrancoed @wujysh The behavior you are seeing is intentional. MSTest framework used to always report the results for data driven tests with mstest.exe with an aggregated result. This was broken with vstest.console.exe for a long time and has been recently fixed. Now both individual result and the aggregated result are reported. The behavior you observed is expected.

Could you please explain the issue you are running into ?

neomarck commented 5 years ago

@dfrancoed @wujysh The behavior you are seeing is intentional. MSTest framework used to always report the results for data driven tests with mstest.exe with an aggregated result. This was broken with vstest.console.exe for a long time and has been recently fixed.

Could you please explain the issue you are running into ?

Ok then, how can we make these aggregate results to work with the Test Results in VSTS Builds? In the image below, it says there were 7 test cases, but actually there are only 4 tests cases, the others are just iterations: image Also, how can we setup VS2017 to get properly displayed these aggregate results in the Test Explorer? Thanks.

wujysh commented 5 years ago

The main problem is that iterations are also counted as tests, and the statistics (total, passed, failed) are not right. It is a new problem encountered after upgrading to MSTest V2. It works fine when I was using MSTest V1 with vstest.console.exe.

@cltshivash It has been recently fixed? Can you tell me how to resolve it? Thanks.

hhotham commented 5 years ago

I have the same issue. Reported test count: VS2015 Test Explorer: 20 Tests TFS2015 Build: 40 Tests

Code: -12 tests without DataRow attributes

I would expect the build to report either 20 tests (as Test Explorer does) or 32 (12 + 18 + 2).

When looking at the results in Test Explorer (within VS) a test that has 2 DataRow attributes gets 3 results. (I have confirmed that the test is only executed twice). I have 8 tests with the DataTestMethod attribute, which correlates with the 8 extra test results the TFS build is displaying.

kaadhina commented 5 years ago

Please provide the following info

  1. Version of VS IDE used in the CI
  2. Is the TFS onprem, if so what version (please also specify which update if you are on an update)
  3. What is the vstest task version
  4. Are you running the tests in CI distributively (Or provide us verbose logs from the task so we can find 3 & 4. To get this logs, set system.debug=true, run the definition and download build logs)
wujysh commented 5 years ago

@kaadhina

  1. Version of VS IDE used in the CI

    VisualStudio version selected for test execution : 15.0

  2. Is the TFS onprem, if so what version (please also specify which update if you are on an update)

    Azure DevOps (Microsoft Internal)

  3. What is the vstest task version

    Microsoft (R) Test Execution Command Line Tool Version 15.0.26929.2

  4. Are you running the tests in CI distributively

    Information: MSTest Executor: Test Parallelization enabled

Please tell me if there is something wrong or other information needed. Thanks.

kaadhina commented 5 years ago

@wujysh Please share verbose build logs. To get this logs, set system.debug=true, run the definition and download build logs. Since you are internal, also consider adding me to your account temporarily (alias same as github handle) and sharing account details and build details to speed things up.

wujysh commented 5 years ago

@kaadhina OK. I will contact you internally later. Thanks.

kaadhina commented 5 years ago

If you are on Azure Devops, this should not happen with the latest agent. Please try this on the latest agent.

acastano1207 commented 5 years ago

Hi Community,

I'm experiencing the same problem:

I have 2 test cases each of them with five scenarios, the resulting output is 10 test cases executed when I only have 2. Does somebody have an idea of what's happening with the implementation of the notation?

I need to execute my test cases locally .

dnlgmzddr commented 5 years ago

Hello everyone, same issue here. Do you guys have any updates on this?

kaadhina commented 5 years ago

Hi, if you are on Azure devops, please update your agents to the latest and try again. This is something that was fixed in the latest agents. If you are on-prem, please share version details

ShreyasRmsft commented 5 years ago

  1. Please get the latest test platform using the vstest platform tool installer

  2. Also use the latest versions of the MsTest Adapter (1.3.2 as of today).

This solution is for anyone on Azure Devops (formerly VSTS) and on TFS 2018.

And on older versions of TFS data driven tests were never officially supported.

If you are on Azure Devops (formerly VSTS) or on TFS 2018 and have followed the above instructions and are still not seeing hierarchical results please set system.debug = true and provide the complete logs zip for investigation and we will re-open the issue.

As for the logging with regards to the console please refer to https://github.com/Microsoft/testfx/issues/484#issuecomment-423464672

neomarck commented 5 years ago
  1. Please get the latest test platform using the vstest platform tool installer
  2. Also use the latest versions of the MsTest Adapter (1.3.2 as of today).

This solution is for anyone on Azure Devops (formerly VSTS) and on TFS 2018.

And on older versions of TFS data driven tests were never officially supported.

If you are on Azure Devops (formerly VSTS) or on TFS 2018 and have followed the above instructions and are still not seeing hierarchical results please set system.debug = true and provide the complete logs zip for investigation and we will re-open the issue.

As for the logging with regards to the console please refer to #484 (comment)

Finally I got the Aggregate tests results in the Build results.

In the Build pipeline > Visual Studio Test step > Test Platform Version: previously I had selected Visual Studio 2015, now I changed it to Visual Studio 2017 and it made the trick, now I can see the tests results in an aggregate manner. At least this solve my issue with the Build results, but the issue remains when you run your test by console. Regards.

elemount commented 5 years ago
  1. Please get the latest test platform using the vstest platform tool installer
  2. Also use the latest versions of the MsTest Adapter (1.3.2 as of today).

This solution is for anyone on Azure Devops (formerly VSTS) and on TFS 2018.

And on older versions of TFS data driven tests were never officially supported.

If you are on Azure Devops (formerly VSTS) or on TFS 2018 and have followed the above instructions and are still not seeing hierarchical results please set system.debug = true and provide the complete logs zip for investigation and we will re-open the issue.

As for the logging with regards to the console please refer to #484 (comment)

@ShreyasRmsft , After take your suggestion, our failed tests have no error and stack traces on the Test blade, it seems it is been hide.

ShreyasRmsft commented 5 years ago

@elemount please send over screen shots and debug logs zip for investigation. (set system.debug = true and upload the entire zip file containing all the logs)

kaadhina commented 5 years ago

The error and stack trace must be visible on clicking the sub results. If this is still not working for you, please share the screen shots and debug logs as asked above.

elemount commented 5 years ago

capture

ShreyasRmsft commented 5 years ago

@elemount and debug logs please.

Also please provide a screen shot where we can see the whole page please or atleast the whole of the tests tab.

elemount commented 5 years ago

capture

elemount commented 5 years ago

ReleaseLogs_65768.zip

elemount commented 5 years ago 
2018-10-22T06:35:28.2476146Z Task         : Visual Studio Test
2018-10-22T06:35:28.2476374Z Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
2018-10-22T06:35:28.2476619Z Version      : 2.141.9
2018-10-22T06:35:28.2476725Z Author       : Microsoft Corporation
2018-10-22T06:35:28.2476868Z Help         : [More information](https://go.microsoft.com/fwlink/?LinkId=835764)
2018-10-22T06:35:28.2477189Z ==============================================================================
2018-10-22T06:35:28.8909285Z SystemVssConnection exists true
2018-10-22T06:35:29.1786397Z SystemVssConnection exists true
2018-10-22T06:35:29.4376202Z In distributed testing flow
2018-10-22T06:35:29.4376382Z ======================================================
2018-10-22T06:35:29.4376644Z Test selector : Test assemblies
2018-10-22T06:35:29.4376836Z Test filter criteria : Priority!=0&TestCategory=MariaDB
2018-10-22T06:35:29.4377036Z Search folder : d:\OrcaSQLAgSEA05\r25\a
2018-10-22T06:35:29.5717529Z Distributed test execution, number of agents in job : 5
2018-10-22T06:35:29.5717886Z Number of test cases per batch : 11
2018-10-22T06:35:29.5718462Z Run settings file : d:\OrcaSQLAgSEA05\r25\a\wasd-test-orcasql-southeastasia1-e.runsettings
2018-10-22T06:35:29.5718891Z Run in parallel : false
2018-10-22T06:35:29.5719061Z Run in isolation : false
2018-10-22T06:35:29.5719264Z Path to custom adapters : d:\OrcaSQLAgSEA05\r25\a\TestAdapters
2018-10-22T06:35:29.5719686Z Other console options : /Platform:x64
2018-10-22T06:35:29.5720002Z ##[warning]Other console options is not supported when using the multi-agent job setting. This option will be ignored.
2018-10-22T06:35:29.5720409Z Code coverage enabled : false
2018-10-22T06:35:29.5720517Z Diagnostics enabled : false
2018-10-22T06:35:29.5795504Z ======================================================
2018-10-22T06:35:29.5796074Z Source filter: d:\OrcaSQLAgSEA05\r25\a\**\*.MySqlaaS.TestCases.*.Codegen.dll
2018-10-22T06:35:31.7111926Z SystemVssConnection exists true
2018-10-22T06:35:31.7141192Z [command]d:\OrcaSQLAgSEA05\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.141.9\Modules\DTAExecutionHost.exe --inputFile d:\OrcaSQLAgSEA05\_temp\input_ac6051e0-d5c4-11e8-8a6c-4116c30b1c84.json
2018-10-22T06:35:31.8018597Z ##########################################################################
2018-10-22T06:35:32.6983474Z ===========================================
2018-10-22T06:35:32.6983690Z AgentName: OrcaSQLAgSEA05-OrcaSQLAgSEA05-5687
2018-10-22T06:35:32.6983888Z ServiceUrl: https://msdata.visualstudio.com/
2018-10-22T06:35:32.6984028Z TestPlatformVersion: 15.9.0
2018-10-22T06:35:32.6984152Z EnvironmentUri: vstest://env/Database Systems/_apis/release/94/65768/82085/3
2018-10-22T06:35:32.6984296Z QueryForTaskIntervalInMilliseconds: 3000
2018-10-22T06:35:32.6984408Z MaxQueryForTaskIntervalInMilliseconds: 10000
2018-10-22T06:35:32.6984520Z QueueNotFoundDelayTimeInMilliseconds: 3000
2018-10-22T06:35:32.6984652Z MaxQueueNotFoundDelayTimeInMilliseconds: 50000
2018-10-22T06:35:32.6984766Z ===========================================
2018-10-22T06:35:33.0533684Z Initializing the Test Execution Engine
2018-10-22T06:35:33.1851137Z Updated Run Settings:
2018-10-22T06:35:33.1851693Z <RunSettings>
2018-10-22T06:35:33.1852015Z   <RunConfiguration>
2018-10-22T06:35:33.1852188Z     <TargetPlatform>x64</TargetPlatform>
2018-10-22T06:35:33.1852349Z     <TargetFrameworkVersion>Framework45</TargetFrameworkVersion>
2018-10-22T06:35:33.1852581Z     <CustomSlicing enabled="true" maxagents="5" numberOfTestCasesPerSlice="11" isTimeBasedSlicing="False" sliceTime="0" />
2018-10-22T06:35:33.1853764Z     <BatchSize>1000</BatchSize>
2018-10-22T06:35:33.1853918Z     <ResultsDirectory>d:\OrcaSQLAgSEA05\_temp\TR_d4012ec5-9c8a-4ca6-ba2c-4d67e70a2fba</ResultsDirectory>
2018-10-22T06:35:33.1854046Z   </RunConfiguration>
2018-10-22T06:35:33.1854143Z   <TestRunParameters>
2018-10-22T06:35:33.1854424Z     <Parameter name="DefaultSubscriptionId" value="ffffffff-ffff-ffff-ffff-ffffffffffff" />
2018-10-22T06:35:33.1857474Z   </TestRunParameters>
2018-10-22T06:35:33.1857567Z   <MSTestV2>
2018-10-22T06:35:33.1857686Z     <CaptureTraceOutput>true</CaptureTraceOutput>
2018-10-22T06:35:33.1857839Z     <EnableBaseClassTestMethodsFromOtherAssemblies>false</EnableBaseClassTestMethodsFromOtherAssemblies>
2018-10-22T06:35:33.1857965Z     <Parallelize>
2018-10-22T06:35:33.1858076Z       <Workers>5</Workers>
2018-10-22T06:35:33.1858188Z       <Scope>MethodLevel</Scope>
2018-10-22T06:35:33.1858288Z     </Parallelize>
2018-10-22T06:35:33.1858380Z   </MSTestV2>
2018-10-22T06:35:33.1858483Z </RunSettings>
2018-10-22T06:35:33.6608889Z Run Settings File Path : C:\Users\sqlbld0\AppData\Local\Temp\tmpDB7E.runsettings
2018-10-22T06:35:34.3331434Z Creating run for selected test assemblies with following parameters
2018-10-22T06:35:34.3331896Z SourceFilter: ThisWontBeUsed TestCaseFilter: Priority!=0&TestCategory=MariaDB
2018-10-22T06:35:34.3332549Z Run title: OrcaSQL MariaDB Scenario Test
2018-10-22T06:35:34.3333109Z test settings id : 51093624
2018-10-22T06:35:34.3333426Z Build location: d:\OrcaSQLAgSEA05\r25\a
2018-10-22T06:35:34.3334054Z Build Id: 1694920
2018-10-22T06:35:35.0695425Z Test run with Id 52582328 associated
2018-10-22T06:35:35.5014769Z Test run '52582328' is in 'InProgress' state.
2018-10-22T06:35:43.4264041Z Slice with id = 31527, of type = 'Discovery' received.
2018-10-22T06:35:43.5802084Z Count of test sources found: 2
2018-10-22T06:35:43.8106357Z =================================================================
2018-10-22T06:35:43.8110495Z Discovering tests from sources
2018-10-22T06:35:43.8764463Z Using new test platform for test execution
2018-10-22T06:35:44.5822809Z ##[warning]DiscoveryMessage : Could not find extensions: d:\OrcaSQLAgSEA05\r25\a\TestAdapters
2018-10-22T06:35:46.1928695Z Number of testcases discovered : 181
2018-10-22T06:35:47.2090998Z Discovered tests 181 from sources
2018-10-22T06:35:47.2091314Z =================================================================
2018-10-22T06:35:48.3439989Z Slice with id = 31528, of type = 'Execution' received.
2018-10-22T06:35:48.3756465Z Count of test sources found: 1
2018-10-22T06:35:48.6133627Z =================================================================
2018-10-22T06:35:48.6133954Z Discovering tests from sources
2018-10-22T06:35:48.6719893Z Using new test platform for test execution
2018-10-22T06:35:49.2888744Z ##[warning]DiscoveryMessage : Could not find extensions: d:\OrcaSQLAgSEA05\r25\a\TestAdapters
2018-10-22T06:35:50.2188604Z Number of testcases discovered : 45
2018-10-22T06:35:50.4589474Z Discovered tests 45 from sources
2018-10-22T06:35:50.4589750Z =================================================================
kaadhina commented 5 years ago

Hi, these are not debug logs. Set variable system.debug = true to collect debug logs, and share the same.

elemount commented 5 years ago

ReleaseLogs_67531.zip capture

elemount commented 5 years ago

Updated the system.debug=true logs

kaadhina commented 5 years ago

What is the version of mstest adapter you are using? Can you please update this is 1.3.2 if it is not already up to date and retry this?

elemount commented 5 years ago

What is the version of mstest adapter you are using? Can you please update this is 1.3.2 if it is not already up to date and retry this?

Hi @kaadhina , we are using mstest adapter 1.3.2 version. Do not need to retry.

kaadhina commented 5 years ago

Is this issue specific to data driven tests or any failing test in general? If it is only for data driven, are you able to see the individual test results for each data row in hierarchical format ?

elemount commented 5 years ago

Is this issue specific to data driven tests or any failing test in general? If it is only for data driven, are you able to see the individual test results for each data row in hierarchical format ?

Hi @kaadhina , it is only for data driven tests. It seems all DataRow is hiden.

parrainc commented 5 years ago

I think this issue #494 marked as bug looks like the one that is mentioned here. cc: @jayaranigarg maybe you have some updates/more info about this that you could share?

cltshivash commented 5 years ago

@elemount Do you see the error logs in the trx file that will be present as an attachment ? In case you have a scaled down repro do share it.

elemount commented 5 years ago

@cltshivash , I can see the error logs in the trx file.

prawalagarwal commented 5 years ago

@elemount Apologies for the delay here. As @cltshivash requested, can you provide us with a scaled down project where you can reproduce this issue. I was not able to repro the same on my side.

elemount commented 5 years ago

Hi @prawalagarwal , Here is the sample code https://shuodl.visualstudio.com/_git/shuodltest?path=%2Fshuodl-test%2FDataRowTest.cs&version=GBmaster .

I got results confuse me. For this project shuodl I got correct result https://shuodl.visualstudio.com/shuodltest/_build/results?buildId=8&view=ms.vss-test-web.test-result-details , but on the same dll on msdata project, I got the unexpected results https://msdata.visualstudio.com/Database%20Systems/_build/results?buildId=1906379&_a=summary&view=ms.vss-test-web.test-result-details .

I think this only happens on https://msdata.visualstudio.com/

prawalagarwal commented 5 years ago

@elemount Can we do a quick skype call? I do not have access to the accounts you mentioned above. If a call works for you please consider scheduling one between 11AM - 8PM IST. You can reach out at praga@microsoft.com with the confirmation.

prawalagarwal commented 5 years ago

Hi @elemount , Any updates here? I checked your run from telemetry and the results were published correctly. The UI should show them properly nested.

prawalagarwal commented 5 years ago

Closing the issue. Please reopen if you need assistance here. Thanks!

slotwinskim commented 4 years ago

Hello, I have the same issue on Visual Studio Code with .Net Core 3.1. I use the newest packages:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0-preview-20200102-01" />
<PackageReference Include="MSTest.TestAdapter" Version="2.1.0-beta2" />
<PackageReference Include="MSTest.TestFramework" Version="2.1.0-beta2" />

MSTest sees bottom code as 3 tests passed. In Visual Studio it's 2, as suppose to.

[DataTestMethod]
[DataRow("abc@def@aa.pl")]
[DataRow("aaaa@aa.pl@")]
t9mike commented 4 years ago

@dfrancoed @wujysh The behavior you are seeing is intentional.

Is there a way to turn this off? I just started experimenting with switching to [DynamicData] for dynamic test cases and see this (I assume similar underpinings?). Getting this "extra" count is very confusing. If 1 or 3 tests fail I see 2 fail, 2 pass.

derekantrican commented 4 years ago

I am still seeing this issue. I'm using the following setup:

I can repro with the same code that is the very first comment on this issue (by just running the test via the Test Explorer in VS). This is very frustrating because I have a couple long tests (~10 min) that take twice as long to run if I use [DataTestMethod] instead of [TestMethod]. (Granted, yes, I can probably do some work to decrease the runtime of my unit tests but this is also obviously a bug that needs to be fixed) Please reopen and fix this issue.

nohwnd commented 4 years ago

Adding for consideration in the next sprint.

aybe commented 4 years ago

As of today, it still happens.

I've upgraded nuget packages but that didn't really help.

Visual Studio still feels it should run a test with an empty parameter passed to DataRowAttribute.

Hylke-Atmos commented 4 years ago

I'd like to argue that this is a bug and undesired behavior!

It seems that not only the results seem off. It also seems like it is incorrectly dealing with the DynamicData method used for retrieving the data. The first entry is retrieved twice (first for the initial call and then second time for actual execution of the data).

Due to this behavior we have a test that is simply failing because its data (a FileStream) is retrieved/opened twice which is (by default) not allowed by the file system.

It is not the most pretty test, but it is a simple and effective test for checking the function that is verifying the version we embed in a file.

public static IEnumerable<object[]> GetFileStreamForGetVersionFromFileStream()
{
    yield return new object[] { new FileStream(@"Assets\temp1.x", FileMode.Open), 0x01 };
    yield return new object[] { new FileStream(@"Assets\temp2.x", FileMode.Open), 0x02 };
    yield return new object[] { new FileStream(@"Assets\temp3.x", FileMode.Open), 0x03 };
}

[DataTestMethod]
[DynamicData(nameof(GetFileStreamForGetVersionFromFileStream), DynamicDataSourceType.Method)]
public void GetVersionFromFileStreamReturnsCorrectValue(FileStream fs, int expected)
{
   MethodInfo method =
       typeof(FileHandler).GetMethod("GetVersionFromFileStream", BindingFlags.NonPublic | BindingFlags.Static);
   var actual = uint.MaxValue;
   if (method != null)
   {
       actual = (uint) method.Invoke(null, new object[] {fs});
   }
   Assert.AreEqual((uint)expected, actual);
   //Cleanup
   fs.Close();
}

This test fails because temp1.x is opened twice and causing the following exception:

The process cannot access the file '...\Assets\temp1.x' because it is being used by another process.

Although it is possible to overcome this particular problem by using the FileShare.Read flag, it is undesired (and unexpected) behavior that MSTest is retrieving and locking the resources twice!

stephenjg24-git commented 4 years ago

I too would agree that the is a bug and needs to be corrected.

pavelhorak commented 3 years ago

AB#844767

vikas0sharma commented 3 years ago

I am still seeing this behavior: "MSTest.TestAdapter" Version="2.1.2" "MSTest.TestFramework" Version="2.1.2" Is it still not fixed.

enisak commented 2 years ago

2.2.8, works totally fine for me