How to send the playwright results to TestRail

[m-sternak]

What would you like the TestRail CLI to be able to do?

GIVEN the playwright tests are executed AND results are available: junit-report.xml, trace.zip, video.webm

I 'am able to send the junit-report content to the TestRail test case result. How to modify the TestRail CLI command in order to send and attach trace and video to TestRail test case result?

Why is this feature necessary on the TestRail CLI?

To have the full report available over TestRail.

More details

Is there any alternative, good practice to store trace, video results and link it to TestRail test case results?

Interested in implementing it yourself?

No

[m-sternak]

Here is the results structure. webm + trace are located in the separate folders.

[jakec-routable]

In order to upload your attachments to your results, the attachments must be associated to the tests results in the junit-report.xml. You will need an additional property in your junit-report.xml - testrail_attachment. For example:

        <testcase classname="TestRail_testsuite1-TestRail" name="test-1-chromium" time="159">
            <properties>
                <property name="testrail_attachment" value="test-results/TestRail_testsuite1-TestRail-test-1-chromium/trace.zip"/>
                <property name="testrail_attachment" value="test-results/TestRail_testsuite1-TestRail-test-1-chromium/video.webm"/>
            </properties>
        </testcase>

To do this, you'll either have to customize the playwright reporter to include these properties or you can modify/edit/rewrite the junit-report.xml to add this information before using trcli.

[m-sternak]

@jakec-routable great thank you. We managed to provide the property manually to the junit-report. It worked. Now, we need to find a way to customize the playwright reported, so it will be injected automatically.

[mjeruzal427]

@m-sternak in order to add annotations automatically to your Playwright junit-report you can use Fixtures and override the default test fixture like this:

// ./fixtures/test.fixture.ts

import { test as base } from '@playwright/test';
import path from 'node:path';
import fs from 'node:fs/promises';

type MyFixtures = {
    testFixture: void;
}

export const test = base.extend<MyFixtures>({

    testFixture: [async ({ }, use, testInfo) => {
        /* BEFORE EACH */

        await use();

        /* AFTER EACH */

        // Attach files to TestRail report on failure
        if (testInfo.status !== testInfo.expectedStatus) {
            const outputDir = path.join(testInfo.outputPath());
            const files = await fs.readdir(outputDir);
            for (const file of files) {
                testInfo.annotations.push({ type: 'testrail_attachment', description: `${outputDir}/${file}` });
            }
        }
    }, { auto: true }],

});

export { expect } from '@playwright/test';

Then in all your spec files you must must replace the "standard" imports:

import {test, expect} from '@playwright/test';

with the new overridden test:

import { test, expect } from '../fixtures/test.fixture';

[acuanico-tr-galt]

The general objective of this issue has been resolved by the other contributors (tysm!), so I’m closing it now; feel free to reopen if needed.