Record each failure reported within one testcase

[jencce]

One testcase can report multiple failures by {tmt,rstrnt,rhts}-report-result helper.

Now the junit reporter only reports one failure message for a failed testcase, no matter how many sub failures reported.

By adding this we can make the result more accurate and convenient for analysis.

[jencce]

For a testsuite, this is important for its failure analysis.

[thrix]

@lukaszachy @happz @psss could you look at this one? it seems to be a Kernel QE blocker:

https://issues.redhat.com/browse/KQE-14

[psss]

Able to reproduce with a simple config:

/plan:
    provision:
        how: container
    discover:
        how: fmf
    execute:
        how: tmt
/test:
    test: |
        tmt-report-result /test/good PASS
        tmt-report-result /test/bad FAIL
        tmt-report-result /test/weird WARN

This results into two tests reported by the display plugin:

report
    how: display
        fail /test
        pass /test
    summary: 1 test passed and 1 test failed

And just one reported by the junit plugin:

<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="1" tests="2" time="0.0">
    <testsuite disabled="0" errors="0" failures="1" name="/plan" skipped="0" tests="2" time="0">
        <testcase name="/test">
            <failure type="failure" message="fail"/>
        </testcase>
        <testcase name="/test"/>
    </testsuite>
</testsuites>

@happz, what's interesting that we get the same serialnumber for both:

  - name: /test
    result: fail
    note:
    log:
      - data/guest/default-0/test-1/output.txt
    serialnumber: 1
    fmf_id:
        name: /test
    ids:
        id:
        extra-nitrate:
        extra-task:
    guest:
        name: default-0
        role:
        __class__:
            module: tmt.result
            name: ResultGuestData
    check: []
    starttime: '2023-09-14T13:58:29.629448+00:00'
    endtime: '2023-09-14T13:58:29.845018+00:00'
    duration: 00:00:00
    __class__:
        module: tmt.result
        name: Result
  - name: /test
    result: pass
    note:
    log:
      - data/guest/default-0/test-1/output.txt
    serialnumber: 1
    fmf_id:
        name: /test
    ids:
        id:
        extra-nitrate:
        extra-task:
    guest:
        name: default-0
        role:
        __class__:
            module: tmt.result
            name: ResultGuestData
    check: []
    starttime: '2023-09-14T13:58:29.629448+00:00'
    endtime: '2023-09-14T13:58:29.845018+00:00'
    duration: 00:00:00
    __class__:
        module: tmt.result
        name: Result

@pfdaly, fyi.

[kkaarreell]

I think it is important to describe and understand the actual use case here. For tmt it is not possible to identify and separate individual failures unless it knows what test framework is used and can process output produced by that specific framework. So this is more about implementing a support for individual frameworks just as it has been proposed by @happz in https://github.com/teemtee/tmt/issues/2355.

Also, you need to make sure that other tools and tester's workflows can work with multiple failure elements. For example the reportportal report plugin won't do it and in RP UI you can specify only a single results review to a single item. I.e. it would be necessary to change how failures are imported to RP.

[lukaszachy]

Related issue is #2389.

I'm starting to think whether tests using tmt-report-result aren't closer to what one can do with 'results:custom'. Shouldn't using tmt-report-result create a 'subtest'?

[psss]

@jencce, could you please check the #2689 pull request to verify that the proposed approach would cover your use case? Thanks!

[kkaarreell]

Just keep in mind that by increasing test case/result granularity you are at the same time making the mapping on the original test case (e.g. for the purpose of import to Polarion) more difficult.

[jencce]

@jencce, could you please check the #2689 pull request to verify that the proposed approach would cover your use case? Thanks!

Thanks for the efforts! Working on it!

[jencce]

Just keep in mind that by increasing test case/result granularity you are at the same time making the mapping on the original test case (e.g. for the purpose of import to Polarion) more difficult.

Thanks for the notice!

[jencce]

@jencce, could you please check the #2689 pull request to verify that the proposed approach would cover your use case? Thanks!

No I didn't see multiple results in the junit report xml with their outputfile contents as messages.

Also if the PATH parameter of rstrnt-report-result contains an ":" as the end of string, error comes up which IMHO should not.

Related issue is #2389.

I'm starting to think whether tests using tmt-report-result aren't closer to what one can do with 'results:custom'. Shouldn't using tmt-report-result create a 'subtest'?

Yeah I think a result for a subtest fits more here. If you report a result with a PATH/name, it's a result of a step inside one testcase. If your PATH/name matches the testcase it self or no PATH/name, it's a result of the whole testcase.

Thanks!

[pfdaly]

Hi @jencce, I've made a few more code pushes to 2689 and I'd like to know does the following meet your requirements.

main.fmf content used for testing

root@fedora:~/2086# cat main.fmf 
/plan:
    provision:
        how: container
    discover:
        how: fmf
    execute:
        how: tmt
    report:
        how: junit
/check:
    test: |
        touch /tmp/pass_log ; echo "Content of pass_log" >> /tmp/pass_log; echo "Content of command exe within /XFSTESTS_START:RedHatEnterprise9:ext4:"; tmt-report-result "/XFSTESTS_START:RedHatEnterprise9:ext4:" PASS -o /tmp/pass_log
        touch /tmp/bad_log ; echo "Content of bad_log" >> /tmp/bad_log; echo "Content of command exe within /test/bad"; tmt-report-result "/test/bad" FAIL -o /tmp/bad_log
        touch /tmp/weird_log ; echo "Content of weird_log" >> /tmp/weird_log; echo "Content of command exe within /test/weird"; tmt-report-result "/test/weird" WARN -o /tmp/weird_log
        touch /tmp/main_additional_log; echo "Content of main_additional_log" >> /tmp/main_additional_log; echo "Content of command exe within /check"; tmt-report-result "" PASS -o /tmp/main_additional_log
    result: custom
root@fedora:~/2086#

Output of the tmt test execution.

root@fedora:~/2086# tmt run -vvv --debug
Using tree '/root/2086'.
/var/tmp/tmt/run-450
tmt version: 1.33.dev9+g6f828393
tmt command line: /usr/bin/tmt run -vvv --debug
Run data not found.
Enabled steps: prepare, discover, execute, finish, report and provision
Workdir '/var/tmp/tmt/run-450/plan' created.
Found 1 plan.

/plan
    Workdir '/var/tmp/tmt/run-450/plan/discover' created.
    Workdir '/var/tmp/tmt/run-450/plan/provision' created.
    Workdir '/var/tmp/tmt/run-450/plan/prepare' created.
    Workdir '/var/tmp/tmt/run-450/plan/execute' created.
    Workdir '/var/tmp/tmt/run-450/plan/report' created.
    Workdir '/var/tmp/tmt/run-450/plan/finish' created.
    discover
    workdir: /var/tmp/tmt/run-450/plan/discover
        how: fmf
        order: 50
        Workdir '/var/tmp/tmt/run-450/plan/discover/default-0' created.
        Git root not found, using '/root/2086.'
        directory: /root/2086
        Copy '/root/2086' to '/var/tmp/tmt/run-450/plan/discover/default-0/tests'.
        Check metadata tree in '/var/tmp/tmt/run-450/plan/discover/default-0/tests'.
        summary: 1 test selected
            /check
    provision
    workdir: /var/tmp/tmt/run-450/plan/provision
        queued provision.provision task #1: default-0

        provision.provision task #1: default-0
        how: container
        order: 50
        primary address: tmt-450-tRfEZThr
        topology address: tmt-450-tRfEZThr
        Workdir '/var/tmp/tmt/run-450/plan/provision/default-0' created.
        Check for container image 'fedora'.
        name: tmt-450-tRfEZThr
        Create network 'tmt-450-network'.
        Start container 'fedora'.
        multihost name: default-0
        arch: x86_64
        distro: Fedora Linux 38 (Container Image)
        kernel: 6.7.4-200.fc39.x86_64
        package manager: dnf
        selinux: yes
        is superuser: yes

        summary: 1 guest provisioned
    prepare
    workdir: /var/tmp/tmt/run-450/plan/prepare
        queued push task #1: push to default-0

        push task #1: push to default-0

        queued prepare task #1: requires on default-0

        prepare task #1: requires on default-0
        how: install
        summary: Install required packages
        name: requires
        order: 70
        where: default-0
        package: 1 package requested
            /usr/bin/flock
            cmd: rpm -q --whatprovides /usr/bin/flock || dnf install -y  /usr/bin/flock
            out: util-linux-core-2.38.1-4.fc38.x86_64

        queued pull task #1: pull from default-0

        pull task #1: pull from default-0

        summary: 1 preparation applied
    execute
    workdir: /var/tmp/tmt/run-450/plan/execute
        queued execute task #1: default-0 on default-0

        execute task #1: default-0 on default-0
        how: tmt
        order: 50
        exit-first: false
        Workdir '/var/tmp/tmt/run-450/plan/execute/default-0' created.
        /check [1/1]
            test: /check
        Execute '/check' as a 'shell' test.
        test wrapper: /var/tmp/tmt/run-450/plan/discover/default-0/tests/tmt-test-wrapper.sh-default-0-default-0
        test topology: /var/tmp/tmt/run-450/plan/execute/data/guest/default-0/check-1/tmt-test-topology.yaml
        test topology: /var/tmp/tmt/run-450/plan/execute/data/guest/default-0/check-1/tmt-test-topology.sh
                cmd:
                    touch /tmp/pass_log ; echo "Content of pass_log" >> /tmp/pass_log; echo "Content of command exe within /XFSTESTS_START:RedHatEnterprise9:ext4:"; tmt-report-result "/XFSTESTS_START:RedHatEnterprise9:ext4:" PASS -o /tmp/pass_log
                    touch /tmp/bad_log ; echo "Content of bad_log" >> /tmp/bad_log; echo "Content of command exe within /test/bad"; tmt-report-result "/test/bad" FAIL -o /tmp/bad_log
                    touch /tmp/weird_log ; echo "Content of weird_log" >> /tmp/weird_log; echo "Content of command exe within /test/weird"; tmt-report-result "/test/weird" WARN -o /tmp/weird_log
                    touch /tmp/main_additional_log; echo "Content of main_additional_log" >> /tmp/main_additional_log; echo "Content of command exe within /check"; tmt-report-result "" PASS -o /tmp/main_additional_log
                out: Content of command exe within /XFSTESTS_START:RedHatEnterprise9:ext4:
                out: Content of command exe within /test/bad
                out: Content of command exe within /test/weird
                out: Content of command exe within /check
        Extract results of '/check'.
                         pass /check/XFSTESTS_START:RedHatEnterprise9:ext4 (on default-0) [1/1]
                         fail /check/test/bad (on default-0) [1/1]
                         warn /check/test/weird (on default-0) [1/1]
                         pass /check/check (on default-0) [1/1]

        summary: 4 tests executed
    report
    workdir: /var/tmp/tmt/run-450/plan/report
        how: junit
        order: 50
        Workdir '/var/tmp/tmt/run-450/plan/report/default-0' created.
        output: /var/tmp/tmt/run-450/plan/report/default-0/junit.xml
        summary: 2 tests passed, 1 test failed and 1 warn
    finish
    workdir: /var/tmp/tmt/run-450/plan/finish

        container: stopped
        container: removed
        Remove network 'tmt-450-network'.
        container: network removed
    Prune '/plan' plan workdir '/var/tmp/tmt/run-450/plan'.
        Workdir '/var/tmp/tmt/run-450/plan/provision/default-0' already exists.
        Workdir '/var/tmp/tmt/run-450/plan/prepare/requires' created.
        Workdir '/var/tmp/tmt/run-450/plan/execute/default-0' already exists.
        summary: 0 tasks completed

total: 2 tests passed, 1 test failed and 1 warn
root@fedora:~/2086#

Generated junit.xml file

root@fedora:~/2086# cat /var/tmp/tmt/run-450/plan/report/default-0/junit.xml
<?xml version="1.0" ?>
<testsuites disabled="0" errors="1" failures="1" tests="4" time="0.0">
    <testsuite disabled="0" errors="1" failures="1" name="/plan" skipped="0" tests="4" time="0">
        <testcase name="/check/XFSTESTS_START:RedHatEnterprise9:ext4">
            <system-out>Content of pass_log
</system-out>
        </testcase>
        <testcase name="/check/test/bad">
            <failure type="failure" message="fail">Content of bad_log
</failure>
            <system-out>Content of bad_log
</system-out>
        </testcase>
        <testcase name="/check/test/weird">
            <error type="error" message="warn">Content of weird_log
</error>
            <system-out>Content of weird_log
</system-out>
        </testcase>
        <testcase name="/check/check">
            <system-out>Content of main_additional_log
</system-out>
        </testcase>
    </testsuite>
</testsuites>
root@fedora:~/2086#

[jencce]

I can't get a similar result like your example is showing: multiple testcase results inside one testsuite.

What I was asking was multiple step results inside one testcase.

With my test case:

$ ls
main.fmf  runtest.sh

$ cat main.fmf 
summary: Test tmt report result
test: bash ./runtest.sh
framework: shell
duration: 10m

$ cat runtest.sh 
#!/bin/bash

l1=$(mktemp)
echo r1 > $l1
l2=$(mktemp)
sleep 2
echo r2 > $l2

echo "one"
tmt-report-result -o $l1 "/TMT:one" FAIL
echo "two"
tmt-report-result -o $l2 "/TMT:two" FAIL
echo "final"
tmt-report-result "" PASS -o $l2

rm -f $l1 $l2

$ tmt test ls .
/filesystems/testmt-report

Installed tmt from tmt-2086 branch, I'm getting result like:

$ cat reports/local-junit.xml
<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="1" tests="1" time="2.0">
    <testsuite disabled="0" errors="0" failures="1" name="/plans/experiment/ttt" skipped="0" tests="1" time="2">
        <testcase name="/filesystems/testmt-report" time="2.000000">
            <failure type="failure" message="fail">one
two
final
</failure>
            <system-out>one
two
final
</system-out>
        </testcase>
    </testsuite>
</testsuites>

This is not what I've expected. I expect something like this:

<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="2" tests="1" time="2.0">
    <testsuite disabled="0" errors="0" failures="2" name="/plans/experiment/ttt" skipped="0" tests="1" time="2">
        <testcase name="/filesystems/testmt-report" time="2.000000">
            <failure type="failure" message="/TMT:one">r1
            </failure>
            <failure type="failure" message="/TMT:two">r2
            </failure>
            <system-out>one
two
final
            </system-out>
        </testcase>
    </testsuite>
</testsuites>

[pfdaly]

So, the reason your output is still reporting only a single test is that the main.fmf did no include the 'result: custom' statement.

I've used the main.fmf on my test environment and included a few additional values to specify the provision and reporting type.

root@fedora:~/2086# cat main.fmf 
/plan:
    provision:
        how: container
    discover:
        how: fmf
    execute:
        how: tmt
    report:
        how: junit
/check:
    result: custom
    summary: Test tmt report result
    test: bash ./runtest.sh
    framework: shell
    duration: 10m
root@fedora:~/2086#

Here is the execution output from tmt where we can see that while only a single runtest.sh file is executed a number of tests are reported.

root@fedora:~/2086# tmt run -vvv --debug
Using tree '/root/2086'.
/var/tmp/tmt/run-454
tmt version: 1.33.dev9+g6f828393
tmt command line: /usr/bin/tmt run -vvv --debug
Run data not found.
Enabled steps: execute, report, finish, prepare, discover and provision
Workdir '/var/tmp/tmt/run-454/plan' created.
Found 1 plan.

/plan
    Workdir '/var/tmp/tmt/run-454/plan/discover' created.
    Workdir '/var/tmp/tmt/run-454/plan/provision' created.
    Workdir '/var/tmp/tmt/run-454/plan/prepare' created.
    Workdir '/var/tmp/tmt/run-454/plan/execute' created.
    Workdir '/var/tmp/tmt/run-454/plan/report' created.
    Workdir '/var/tmp/tmt/run-454/plan/finish' created.
    discover
    workdir: /var/tmp/tmt/run-454/plan/discover
        how: fmf
        order: 50
        Workdir '/var/tmp/tmt/run-454/plan/discover/default-0' created.
        Git root not found, using '/root/2086.'
        directory: /root/2086
        Copy '/root/2086' to '/var/tmp/tmt/run-454/plan/discover/default-0/tests'.
        Check metadata tree in '/var/tmp/tmt/run-454/plan/discover/default-0/tests'.
        summary: 1 test selected
            /check
    provision
    workdir: /var/tmp/tmt/run-454/plan/provision
        queued provision.provision task #1: default-0

        provision.provision task #1: default-0
        how: container
        order: 50
        primary address: tmt-454-DPKOZMWM
        topology address: tmt-454-DPKOZMWM
        Workdir '/var/tmp/tmt/run-454/plan/provision/default-0' created.
        Check for container image 'fedora'.
        name: tmt-454-DPKOZMWM
        Create network 'tmt-454-network'.
        Start container 'fedora'.
        multihost name: default-0
        arch: x86_64
        distro: Fedora Linux 38 (Container Image)
        kernel: 6.7.4-200.fc39.x86_64
        package manager: dnf
        selinux: yes
        is superuser: yes

        summary: 1 guest provisioned
    prepare
    workdir: /var/tmp/tmt/run-454/plan/prepare
        queued push task #1: push to default-0

        push task #1: push to default-0

        queued prepare task #1: requires on default-0

        prepare task #1: requires on default-0
        how: install
        summary: Install required packages
        name: requires
        order: 70
        where: default-0
        package: 1 package requested
            /usr/bin/flock
            cmd: rpm -q --whatprovides /usr/bin/flock || dnf install -y  /usr/bin/flock
            out: util-linux-core-2.38.1-4.fc38.x86_64

        queued pull task #1: pull from default-0

        pull task #1: pull from default-0

        summary: 1 preparation applied
    execute
    workdir: /var/tmp/tmt/run-454/plan/execute
        queued execute task #1: default-0 on default-0

        execute task #1: default-0 on default-0
        how: tmt
        order: 50
        exit-first: false
        Workdir '/var/tmp/tmt/run-454/plan/execute/default-0' created.
        /check [1/1]
            test: Test tmt report result
        Execute '/check' as a 'shell' test.
        test wrapper: /var/tmp/tmt/run-454/plan/discover/default-0/tests/tmt-test-wrapper.sh-default-0-default-0
        test topology: /var/tmp/tmt/run-454/plan/execute/data/guest/default-0/check-1/tmt-test-topology.yaml
        test topology: /var/tmp/tmt/run-454/plan/execute/data/guest/default-0/check-1/tmt-test-topology.sh
                cmd: bash ./runtest.sh
                out: one
                out: two
                out: final
        Extract results of '/check'.
                         fail /check/TMT:one (on default-0) [1/1]
                         fail /check/TMT:two (on default-0) [1/1]
                         pass /check/check (on default-0) [1/1]

        summary: 3 tests executed
    report
    workdir: /var/tmp/tmt/run-454/plan/report
        how: junit
        order: 50
        Workdir '/var/tmp/tmt/run-454/plan/report/default-0' created.
        output: /var/tmp/tmt/run-454/plan/report/default-0/junit.xml
        summary: 1 test passed and 2 tests failed
    finish
    workdir: /var/tmp/tmt/run-454/plan/finish

        container: stopped
        container: removed
        Remove network 'tmt-454-network'.
        container: network removed
    Prune '/plan' plan workdir '/var/tmp/tmt/run-454/plan'.
        Workdir '/var/tmp/tmt/run-454/plan/provision/default-0' already exists.
        Workdir '/var/tmp/tmt/run-454/plan/prepare/requires' created.
        Workdir '/var/tmp/tmt/run-454/plan/execute/default-0' already exists.
        summary: 0 tasks completed

total: 1 test passed and 2 tests failed
root@fedora:~/2086# 

With the inclusion of the 'result: custom' each execution of the tmt-report-result is treated as an independent test and thus reported as such, as can be viewed the generated junit report.

root@fedora:~/2086# 
root@fedora:~/2086# cat /var/tmp/tmt/run-454/plan/report/default-0/junit.xml
<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="2" tests="3" time="0.0">
    <testsuite disabled="0" errors="0" failures="2" name="/plan" skipped="0" tests="3" time="0">
        <testcase name="/check/TMT:one">
            <failure type="failure" message="fail">r1
</failure>
            <system-out>r1
</system-out>
        </testcase>
        <testcase name="/check/TMT:two">
            <failure type="failure" message="fail">r2
</failure>
            <system-out>r2
</system-out>
        </testcase>
        <testcase name="/check/check">
            <system-out>r2
</system-out>
        </testcase>
    </testsuite>
</testsuites>
root@fedora:~/2086#

[jencce]

Oh the "result: custom" needs to be in the testcase main.fmf. I was putting it in the plan.fmf.

Yes, this is what I need! Thanks very much for doing this!

I got the the custom junit.xml and uploaded it to ReportPortal manually, then tfacon did can analysis it.

Two more issues: 1, I've noticed that the 2086 branch is behind main branch much, maybe that's why I failed to use the plugin to report to ReportPortal automatically. I'll try rebasing 2086 to main to test if this is the reason or it's my bad setup.

2, The way you are writing the main.fmf interested me a lot. There are a /check and a /plan part, which I can't find in the tmt doc to elaborate more about them. Is it a testcase or a plan ? I want to learn this magic :) Thanks!

[pfdaly]

I can't recall where I first observed the plan and test metadata being placed inside the same .fmf file. It was probably a couple of years ago when the effort to update existing RHEL test cases for execution with tmt began. Anyhow, it appears to work just as if they were separate files, though I'm unsure of whether it's actually supported.

I've seen cases where a single .fmf file is used to define the configuration for mutiple tests, with test configuration residing below that object. eg.

/test_1
    summary: 
    description:

/test_2
    summary: 
    description:

I thought that given that new properties seemed to be read from each / entry it should possible to also include a /plan configuration for the plan, and that appears to be true.

As for ReportPortal or Polarion, I've not utilised that functionality, do let me know if there are any issues, though I may have to confer with the other members of the tmt team as I've little knowledge about them.

I've to add an entry to the release not for this functionality, and will certainly rebase the merge request to include the last code after that.

[jencce]

Thanks for the detailed explanation! It's very helpful to me.

Can we make the "result: custom" config effective in the plan instead of testcase ? In this case, we only need one line in the plan.fmf file while not updating nearly all of the testcases.

[kkaarreell]

Can we make the "result: custom" config effective in the plan instead of testcase ? In this case, we only need one line in the plan.fmf file while not updating nearly all of the testcases.

You can use fmf inheritance a drop it in main.fmf stored higher in the directory tree.

[jencce]

Can we make the "result: custom" config effective in the plan instead of testcase ? In this case, we only need one line in the plan.fmf file while not updating nearly all of the testcases.

You can use fmf inheritance a drop it in main.fmf stored higher in the directory tree.

Thanks Karel, I'll try this.

[jencce]

@pfdaly Hmm.. with 2086 patches, I got this error when tmt reporting directly to ReportPortal:

plan failed
The exception was caused by 1 earlier exceptions
Cause number 1:
    Received non-ok status code from ReportPortal: {"errorCode":4001,"message":"Incorrect Request. Invalid JSON input: Cannot deserialize value of type `java.util.Date` from String \"2024-07-10T20:10:34,352572687-04:00\": not a valid representation (error: Failed to parse Date value '2024-07-10T20:10:34,352572687-04:00': Cannot parse date \"2024-07-10T20:10:34,352572687-04:00\": while it seems to fit format 'yyyy-MM-dd'T'HH:mm:ss.SSSZ', parsing fails (leniency? null)); nested exception is com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `java.util.Date` from String \"2024-07-10T20:10:34,352572687-04:00\": not a valid representation (error: Failed to parse Date value '2024-07-10T20:10:34,352572687-04:00': Cannot parse date \"2024-07-10T20:10:34,352572687-04:00\": while it seems to fit format 'yyyy-MM-dd'T'HH:mm:ss.SSSZ', parsing fails (leniency? null))\n at [Source: (PushbackInputStream); line: 1, column: 152] (through reference chain: com.epam.ta.reportportal.ws.model.log.SaveLogRQ[\"time\"])"}

Looks like it relates to \"$(date -Ins)\" in tmt-2086 commit b938e067.

At the same running plan, the generated junit xml can be uploaded to ReportPortal manually without such time issue.

[pfdaly]

Looks like it relates to "$(date -Ins)" in tmt-2086 commit b938e06.

I've not tested the uploading to ReportPortal, but I've updated the date command format to match the results returned from the code from the main branch for a non-restraint test execution. I'm hopeful this will rectify the issue.

[jencce]

Can we make the "result: custom" config effective in the plan instead of testcase ? In this case, we only need one line in the plan.fmf file while not updating nearly all of the testcases.

You can use fmf inheritance a drop it in main.fmf stored higher in the directory tree.

Works fine! Thanks very much!

[jencce]

Looks like it relates to "$(date -Ins)" in tmt-2086 commit b938e06.

I've not tested the uploading to ReportPortal, but I've updated the date command format to match the results returned from the code from the main branch for a non-restraint test execution. I'm hopeful this will rectify the issue.

Yes, it does. The time format error has gone. Generated junit.xml file looks fine, can be uploaded to ReportPortal manually and can be analyzed by tfacan(AI).

But when reporting directly to ReportPortal, the result does not look like a custom result at all. Only the first reported result has been recorded and the whole test case result is "Skipped" which is odd because it's not used in the testcase.

Given the issue is filed for report:junit, I'd say the work is done great. If it requires more digging, the ReportPortal issue should be addressed in another issue.

Thanks!