Use datatype as testsuite name instead of instance path

[philippleidig]

Issue

In some cases it is necessary to use the data type of the testsuite instead of the internal instance path in order to be able to assign the test results to the previously parsed test suites afterwards. For example, if the *.plcproj file is parsed to discover test-suites / test-cases, only the data type of the testsuite is present here. The instance path is only available at runtime.

Instance path: MAIN.AxisGeneric_Tests_Instance Datatype: AxisGeneric_Tests

PROGRAM MAIN
VAR
    AxisGeneric_Tests_Instance : AxisGeneric_Tests;
    DriveGeneric_Tests_Instance : DriveGeneric_Tests;
END_VAR

tcunit_testresults.xml

<testsuite id="0" name="MAIN.AxisGeneric_Tests_Instance" tests="2" failures="1">
    <testcase name="Should_Fail_When_AxisIdIsZero" classname="AxisGeneric_Tests_Instance" status="PASS"></testcase>
</testsuite>

Switching between data type and instance path when generating the name of the execution definition and the test cases would be useful.

Solution

Define a specific parameter for keeping backwards compatibility

tcunit_testresults.xml

<testsuite id="0" name="AxisGeneric_Tests" tests="2" failures="1">
    <testcase name="Should_Fail_When_AxisIdIsZero" classname="AxisGeneric_Tests" status="PASS"></testcase>
</testsuite>

[philippleidig]

see TcUnit.TestAdapter for more details

[sagatowski]

Hey!

This is not clear to me. The "UseDataTypesAsTestNames" sound like the test-names should differ? But the testcase in the end-report is in both cases Should_Fail_When_AxisIdIsZero?

[philippleidig]

Hey @sagatowski :)

sorry that I have not described this in detail. It's about the names of the test suites, not the test cases. Therefore, "UseDataTypeAsTestSuiteName" would probably be a much better solution.

In TcUnit version 1.2.0.0, the name of the test suite is read via the instance path with the attribute {attribute 'instance-path'}. This information is only available online in RUN mode.

TcUnit-VsTestAdapter and other tools require the offline information of the test suites to define the names. Only the data type or name of the function block is available in the *.plcproj file. The instance path of the function block instance cannot be accessed offline.

E.g. the TcUnit-VsExtension project aims to integrate "TcUnit" into the Visual Studio Test Explorer. This would require pure offline information about the test suites. This would enable the unit tests to be executed automatically via "vstest" or "TcUnit-VsTestAdapter". For example, also automatically after a rebuild of the solution.

[sagatowski]

Hey @philippleidig !

Aha! Now I understand what you want to achieve! First of all; Those are two really great tools. I should look deeper into them, but I'm already very impressed with the VsExtension, that is insanely useful and would make TcUnit feel much more integrated and in line with what other unit testing frameworks from the IT-world have to offer. I love this project already.

I'm just quickly looking at the code. It's one thing I need you to help me understand. Why can you only use the *.plcproj-file to figure out the information you need? Isn't it possible to get the instance-path? I mean, even though it is generated at run-time (through the reflection), the complete test-path is available for all the tests through the test-task->the testprogram->the test suite instances, no?

[philippleidig]

@sagatowski, I'm glad to hear that :) When a first stable version is available, both tools can also be further developed under the TcUnit organization (by you). I would also like to provide a setup like Zeugwerk Twinpack does, instead of using the VSIX.

Unfortunately, I have not yet found an elegant way to read the instance paths / instance names of the test suites offline. From my point of view, it is neccesary to start at the unit test PLC task -> PLC PROGRAM-> parse for function blocks -> build name / instance path + read data type -> read the file path to the .TcPOU file via .plcproj -> read *.TcPOU as XML file and parse for "EXTENDS TcUnit.FB_TestSuite"

The PLC PROGRAM would probably also have to be parsed recursively, as each user can freely define the structure of the unit testing project.

Offline via the .plcproj you would only have to search for .TcPOU files in the *.plcproj file and check the inheritance of "EXTENDS TcUnit.FB_TestSuite" and thus filter for test suites.

The big disadvantage of using the data types as TestSuite names is the use of "Tc3_JsonXml". This library is only included as of 4022.0. TcUnit version 1.2.0.0 is currently compatible back to build 4018 / 4020 or so. But perhaps this would also be a good time to cut off old habits :)

[dfreiberger]

@philippleidig you probably already considered this, but if the project has been compiled the TMC file may have the information you need. For example here is Python code to extract TestSuites in a project (using https://github.com/tcunit/ExampleProjects/tree/master/AdvancedExampleProject).

from lxml import etree as ET

tmc = ET.parse('IOLink.tmc')

# Find all data types which extend FB_TestSuite
# <DataType>
#     <Name>FB_DiagnosticMessageDiagnosticCodeParser_Test</Name>
#     <BitSize>33561984</BitSize>
#     <ExtendsType Namespace="TcUnit">FB_TestSuite</ExtendsType>
#     <Method>
#         <Name>WhenProfileSpecificExpectProfileSpecific</Name>
#         <Local>
test_suites = tmc.xpath('//DataType/ExtendsType[text()="FB_TestSuite"]/../Name/text()')

# Find all symbols where base type matches the FB_TestSuites found above
# <Symbol>
#         <Name>PRG_TEST.fbDiagnosticMessageTimeStampParser_Test</Name>
#         <BitSize>33561984</BitSize>
#         <BaseType>FB_DiagnosticMessageTimeStampParser_Test</BaseType>
#         <BitOffs>767816192</BitOffs>
#  </Symbol>
for test_suite in test_suites:
    symbol_path = tmc.xpath('//Symbol[BaseType="{}"]/Name/text()'.format(test_suite))
    print(symbol_path[0], ':', test_suite)

Result

~\source\repos\ExampleProjects\AdvancedExampleProject\IOLink\IOLink [master ≡ +2 ~0 -0 !]> python .\test.py
PRG_TEST.fbDiagnosticMessageDiagnosticCodeParser_Test : FB_DiagnosticMessageDiagnosticCodeParser_Test
PRG_TEST.fbDiagnosticMessageFlagsParser_Test : FB_DiagnosticMessageFlagsParser_Test
PRG_TEST.fbDiagnosticMessageParser_Test : FB_DiagnosticMessageParser_Test
PRG_TEST.fbDiagnosticMessageTextIdentityParser_Test : FB_DiagnosticMessageTextIdentityParser_Test
PRG_TEST.fbDiagnosticMessageTimeStampParser_Test : FB_DiagnosticMessageTimeStampParser_Test

Edit: I guess this doesn't handle cases where FB_TestSuite instances exist inside of FBs or not at the root level, in this case you would need to resort to recursively parsing the datatype tree as you mentioned. This still might be manageable since it should be contained in this single file, but not ideal.

[sagatowski]

@dfreiberger I like this idea! I haven't seen any instances of testsuites that reside inside other FB's out in the wild (as the documentation and all examples create them in the root-level), but as you correctly stated, it is technically possible to do so.

[philippleidig]

@dfreiberger, to be honest, I hadn't even considered it before. Thanks for pointing this out. I will look at this in detail in the next few days. Looks pretty straight forward in first place

[sagatowski]

@philippleidig I've finally had some time to look into what you've done, and I like this! My plan is to release 1.3 in a few weeks. It would be fantastic if we could release this as a tool for version 1.4 of TcUnit. It's definitely worth as a lot of the users of TcUnit are not happy with the "loose" integration of TcUnit into TcXaeShell/VS right now. The people that have experience with tdd/unit testing from the IT-world expect tighter integration into the IDE, and it feels like what you've been working on would fit that spot. Probably won't have time to dive deep into this after christmas, but I'd like to discuss what you've done over a Teams-meeting if possible. There are many other things that would have to be there for a release that I've realized people expect more or less. Like good documentation with examples. I've moved most of the documentation from the wordpress site over to GitHub now (so it's just MD-files) so it's possible to collaborate on that stuff as well. Anyway, you're welcome to drop me a message (in whatever channel you find suitable).

Either case, I think integration into TcXaeShell/VS is really neat, and I hope the journey there is not too long! (though I of course assume it will be all than trivial to get this to be reliable/stable).

Thanks for your hard work!

[dfreiberger]

@philippleidig I spent a bit of time to re-work your adapter to use the TMC file and also to parse test cases via TEST('test_name') tokens rather than relying on the method names. See an example here: https://github.com/dfreiberger/TcUnit-VsTestAdapter/blob/dfreiberger/parse-tmc/tests/TcUnit.TestAdapter.Tests/TestRunnerTests.cs which finds the tests in this project: https://github.com/dfreiberger/TcUnit-VsTestAdapter/blob/dfreiberger/parse-tmc/tests/TcUnit.TestAdapter.Tests/PlcTestProject/FirstPLC/POUs/FB_TestSuite1.TcPOU

This was done in a hurry so I am sure it needs a lot of work, but I just wanted to try out the general concept.

var filePath = @"PlcTestProject\PlcTestProject.tsproj";

var project = TwinCATXAEProject.Load(filePath);
var logger = Mock.Of<IMessageLogger>();
var testRunner = new TestRunner();
var testCases = testRunner.DiscoverTests(project, logger);

List<string> testCaseNames = new List<string>()
{ "PRG_TESTS.fbTestSuite1Instance1.TestCase1A",
  "PRG_TESTS.fbTestSuite1Instance1.TestCase1B",
  "PRG_TESTS.fbTestSuite1Instance1.TestCase1C",
  "PRG_TESTS.fbTestSuite1Instance1.TestCase1D",
  "PRG_TESTS.fbTestSuite2Instance1.TestCase2A",
  "PRG_TESTS.fbTestSuite2Instance1.TestCase2B",
  "PRG_TESTS.fbTestSuite2Instance1.TestCase2C",
  "PRG_TESTS.fbTestSuite2Instance2.TestCase2A",
  "PRG_TESTS.fbTestSuite2Instance2.TestCase2B",
  "PRG_TESTS.fbTestSuite2Instance2.TestCase2C",
};

Assert.IsTrue(testCases.Count() == testCaseNames.Count);
foreach (var testCase in testCases)
{
    Assert.IsTrue(testCaseNames.Contains(testCase.FullyQualifiedName));
}

[philippleidig]

Closed since @dfreiberger's solution with parsing the PLC TMC file works like a charm. See PR for more information.