Generated trx does not contain test categories of native tests

[grondinjc]

When launching vstest.console.exe with the logger type trx, tests categories defines on test methods are extracted into the trx for managed tests but not for the natives ones.

Steps to reproduce

I made an isolated solution that contains both a c# and a c++ project that have unit tests. To reproduce my setup, open the solution, compile, run the script trx_generator.bat and analyse the generated trx file in the TestResults folder.

Note that the c++ project was taken directly from the test assets of this repo. vstest_issue_testcategory.zip

Expected behavior

I would have expected the test categories to be extracted into the trx just like the managed behavior.

Actual behavior

As of today, few attributes are considered from c++ projects. These are the one I noticed :

• TEST_IGNORE
• TEST_OWNER(L"grondinjc")
• TEST_PRIORITY(2)

Diagnostic logs

My environment uses vstest from the 2017 enterprise VisualStudio installation (Microsoft (R) Test Execution Command Line Tool Version 15.7.2) on Windows10. As per the How-to, here are the diagnostics collected when running vstest :

issue_log_vstest2017enterprise.txt

[kaadhina]

@grondinjc There is an attribute called TestCategory for C#, but I don't think there is an equivalent attribute for cpp though. Neither does your sample use any such attribute for the cpp project. Can you mention the exact attribute that you don't see reported in the trx?

[grondinjc]

Hi @kaadhina, I am well aware of the c# attribute TestCategory (from namespace Microsoft.VisualStudio.TestTools.UnitTesting). Indeed, there are no identical equivalent in c++ because the language does not support decorated methods the same way. However, the equivalent is done using macros.

As shown in many example (see vstest_issue_testcategory.zip from my initial post), the TEST_METHOD_ATTRIBUTE macro is used (see vstest_issue_testcategory.zip -> CPPSimpleProj -> unittest1.cpp) The macro is very simple, it allows to define a key value pair using wstrings.

Having the native test being decorated-ish, I would have expected to have them exported in the trx just like it is done in c#. To be more precise, from my above snapshot, the native test has a macro that exposes the key SomeNativeAttributeKey and the value SomeNativeAttributeValue. It exposes another category pair for which the key is AnotherNativeAttributeKey and the value is AnotherNativeAttributeValue. I would have then expected the native attributes to be exposed in the trx in such manner :

In order to see the actual generated trx file, see file vstest_issue_testcategory.zip -> vstest_issuetestcategory -> TestResults > grondinjc_output_sample.trx. You will be able to see how the categories are exposed for c# and c++. In my demo app, the c# test is called SomeManagedTest and the c++ test is called ANativeTest.

Thanks

[kaadhina]

Okay, I understand the ask. I will mark this as enhancement and add this ask to our backlog. But please note that this is not a priority for us right now, since we have not had any other asks. Can you also please specify your complete scenario and why this is important for you?

[grondinjc]

@kaadhina This is needed for a continuous integration system that is integrated within VSTS that ends up using MTM. Those attributes are then use to categorize test methods in the proper test cases and test suites.

Having that feature, a complete scenario usage would be :

1. Decorate test methods with metadata specific to my usecases
2. A release definition is trigged using VSTS release management
3. Run tests (native and managed combined) with vstest.console argument /Logger:trx
4. Run custom app that consume the trx file, creates test suites and test cases according to test metadata from the trx and publishes test results on test cases.

If I had no native tests, I would have done something using reflection to find my metadata directly. However, this cannot be done easily with native code. Therefore, the trx seems to be the best common denominator for me. Voila :)

[kaadhina]

Thank you. Right now, the only workaround I can think of is to add some sort of a prefix to test method name to identify the category, such as TestCategory_TestMethodName. I will get this added to the backlog though. Thanks for the feedback!

[csoltenborn]

Just in case you would be willing to port your C++ tests to google test: You might be able to make use of traits to decorate your tests and then run selections of the tests using test case filters. You would end up with as many invocations of VsTest.Console as you have categories, though.

[JunielKatarn]

I'm running into a like related error. (See https://github.com/JunielKatarn/VSTestSample/blob/master/SampleTest/TestClass2.cpp#L19)

        BEGIN_TEST_METHOD_ATTRIBUTE(TestMethod1)
            TEST_OWNER(L"FailsIntermittently")             // Recognized
            TEST_PRIORITY(1)                               // Recognized
            TEST_METHOD_ATTRIBUTE(L"TestCategory", L"Cat") // Ignored
            TEST_DESCRIPTION(L"MayFail")                   // Ignored
        END_TEST_METHOD_ATTRIBUTE()

If I invoke:

vstest.host.exe .\x64\Debug\SampleTest.dll /TestCaseFilter:Owner=FailsIntermittently

The test labeled with that Owner attribute gets run successfully.

But, if I invoke:

vstest.host.exe .\x64\Debug\SampleTest.dll /TestCaseFilter:TestCategory=Cat

The test is not located. Is this the same root issue?

If this is not a priority for the development team, can someone maybe point me to where to start looking to make a fix and contribute it myself?

[grondinjc]

@JunielKatarn Yes, this is the same issue that I have. So far, my workaround is the fudge more information into the owner field since it is a considered string using an user-defined delimiter. To apply a testcasefilter on top of it, I then use the contains operations ~. Because the string values can overlap when using a string contains, I added a prefix.

BEGIN_TEST_METHOD_ATTRIBUTE(TestMethod1)
            TEST_OWNER(L"FailsIntermittently|Animal: Cat")             // Recognized
            TEST_PRIORITY(1)                               // Recognized
        END_TEST_METHOD_ATTRIBUTE()

Invocation then looks like that vstest.host.exe .\x64\Debug\SampleTest.dll /TestCaseFilter:"Owner~Animal: Cat"

[TheHubbit]

I ran into the same issues. Why is this considered an enhancement and not a bug though?

Should be obvious already, but just to make very clear:

BEGIN_TEST_METHOD_ATTRIBUTE(AbcTest)
    TEST_METHOD_ATTRIBUTE(L"Owner", L"abc")
    TEST_METHOD_ATTRIBUTE(L"TestCategory", L"abc")
END_TEST_METHOD_ATTRIBUTE()
    TEST_METHOD(AbcTest){  }

Works: vstest.console /TestCaseFilter:"Owner=abc" AbcTest.dll Fails: vstest.console /TestCaseFilter:"TestCategory=abc" AbcTest.dll

[puttagunta]

I'm not sure, if I need to create a new issue for the problem described below. Trx files generated for xunit adapter are not recording any Trait attributes. It works fine in mstest where the category attributes are recorded with TestCategory tag under TestDefinitions. Is this also an enhancement or a bug? Please advise if I need to create a new bug for this?

[TheHubbit]

I'm not sure, if I need to create a new issue for the problem described below. Trx files generated for xunit adapter are not recording any Trait attributes. It works fine in mstest where the category attributes are recorded with TestCategory tag under TestDefinitions. Is this also an enhancement or a bug? Please advise if I need to create a new bug for this?

I agree, creating a new bug for this (attributes not fully working in unmanaged tests) might be better as the title doesn't seem to be related.

[nohwnd]

This is a new feature and won't be implemented, we are focusing on adding new features to Testing.Platform instead. https://aka.ms/testingplatform