Feature Request - JUnit Report System-Out

[felipefoz]

Idea

Hi,

I would like to propose a feature regarding the JUnit Report. In Gitlab, in case of failure, you can attach something as system out tag.

https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html

My idea is that during the report, in case of a failure we would create a screenshot of the test VI and use the system out tag to point to that file, so we could visually inspect the offending test in the Gitlab Web.

Code for getting the VI BD:

Use Case: CI workflow When: Test Failure

Example

Gitlab View

JUnit Report Example

<?xml version="1.0" encoding="UTF-8"?>
<testsuites framework-name="Caraya" framework-version="1.4.2.145">
<testsuite name="Test case" errors="0" skipped="0" tests="1" failures="1" timestamp="2023-09-13T21:21:13.680+01:00">
    <testcase classname="Test case" name="First Assertion Name" time="1.625275">
         <failure message="test failure">"FAIL"</failure>
        <system-out>[[ATTACHMENT|test.png]]</system-out>
    </testcase>
</testsuite>
</testsuites>

To Decide

1. Where to put? I assume somewhere where the report is being generate, development time usually is not using reports.
2. When to trigger? Only when report is being generated.
3. What is the filename for the screenshot? It could be any name, but using the VI name + something could be useful, especially in Gitlab, where you have to specify this file to be included as artifact (my idea is that you could use some wild card, *_failuresnippet.png".

Anyone finds this useful?

Regards,

[francois-normandin]

Hi @felipefoz

This would not require any changes to Caraya. It is already possible to extend the framework in two different ways to achieve this goal, And I'd personally argue against this in the base framework as it is an time-expensive operation that is only suitable for certain CI workflows. To maintain the framework fast and lightweight, I hope one of the approaches below would fit the community's needs!

First approach: When your Test Suite finishes its execution, you can extract the Call Chain or all Tests that failed. From there, your CI pipeline could rebuild the VI Path from the project root, and script the screenshot of the failed Test VI.

Second approach: It is more elegant, but requires that you build your own suite of wrappers around Caraya.

You could use the Test or Assert Properties. While the Test is executing, every time you encounter a failure, you reconstruct the path to the VI (easy, it is still in memory at this point), generate the screenshot and add the path of your PNG file into the properties. This property will be in the report, which you can extract in your CI pipeline.

The best thing with the Wrapper is that you can customize for your company`s needs at will.

Here is an example (attached code LV2021) with a single Test VI.

caraya wrapper LV2021.zip

This is the Wrapper function prototype for "Assert Equal Variant"

Interactive window shows the results (same info as in the JUnit file for your Test Suite)

Junit results produced:

All screenshot paths are readily available in the Extended Test Results.

Disclaimer: There is no error handling in this Assert Property wrapper example.

[felipefoz]

Hi @francois-normandin,

I forgot to reply this one. Sorry about that.

Although your solution does the job of inserting a reference to the BD file in the report, it does not satisfy the requirement for the JUnit report as described by Gitlab.

the <system-out> tag should be under testcase tag, and not under the properties tag.

Considering the first approach you mentioned, I can assume it is not possible to do this as part of the Caraya. So the solution would be to parse the "Test Results" cluster and the XML to include this tag in the right place. Am I right?

Regards,

[francois-normandin]

@felipefoz You could overwrite the JUnit Report class and insert the system-out anywhere you need. In conjunction with the above-mentioned examples, where we use the default JUnit format, you would intercept at the point where the report is generated, extract the PNG path, and place it where you want in the XML output file.

Do you think that would work?

Ref: https://github.com/JKISoftware/Caraya/wiki/Creating-Custom-Test-Reports

[francois-normandin]

We might want to support a JUnit.Gitlab ???

[felipefoz]

Hum, good point, why not? I will take a look on how to do it.

If I do it, I will make sure to update the post here with a link to this custom report.