Automatic Report Merging for Sharded Automation Tests

[itz4blitz]

Describe the solution you'd like We use Playwright. Specifically, we would like to use the Sharding feature to decrease our test execution time:

https://playwright.dev/docs/test-sharding

When using sharding, the standard report for Playwright must be merged as it generates a separate report for each shard. We would like to do the same with ReportPortal. I've seen it is possible to manually merge launch reports, so I assume it should be possible to automate this.

Is your feature request related to a problem? Please describe.

If we use the sharding feature in Playwright, we are left with 2 separate and must manually merge them.

[DzmitryHumianiuk]

@itz4blitz practically we do have capabilities for this. @AmsterGet i'm not sure if we have it already added into the Playwright agent. But it definitely something which we already tried to resolve for Cypress as well, and everything we need is added on the level of client-js SDK

[AmsterGet]

Hello @itz4blitz ! Since there are independent runs on different machines, Playwright does not provide an orchestration mechanism for their results to external reporters. From our side, I can offer two options:

1. Trigger a launch before all tests manually via the API, then specify its id in the reporter config for each job. As a run post-step (when all tests finished), launch also needs to be finished manually.

2. Specify a unique CI run ID as a launch attribute, which will be the same for different jobs in the same run (this could be a commit hash or something else). As a follow up step, you can use the API to filter the required launches by the provided attribute, collect their ids, and then simply call the "merge" launch API endpoint to combine the launches into one.

Please let me know if there are any questions.

Related to https://github.com/reportportal/agent-js-playwright/issues/86

[Lazar-Adrian]

Hello,

Playwright is supporting an orchestration mechanism for their reporting starting from version 1.37 Will this help extending the functionality for the playwright agent as well?

[AmsterGet]

Hi @Lazar-Adrian ! Thanks for your comment.

I checked the Playwright feature you mentioned and it is not applicable in our case. The CLI tool merge-reports is designed to merge local reports represented by files in the file system, so it is not suitable for external reporting systems like ReportPortal, as it requires at least network communication through the right endpoints.

In general the approach looks very similar to what I've described in the comment above. Would it be useful if we created some service functions/CLIs that perform the actions described in my options? This would still require some additional steps:

• For example, adding two additional steps to your CI pipelines for option 1;
• or providing a CI_BUILD_ID and a final merge step for option 2.

Please let us know what you think about it :)

[Lazar-Adrian]

Hi @AmsterGet Thank you for your response.

We managed to implement the approach that you described at step 1 with an API trigger of launch and an additional job to finish it. Regarding your question of adding some service functions/CLIs, it would be an improvement for the ease of use, although it can be done without any difficulties at the current moment.

[markwhitfeld]

So, just to add another scenario here.

Our current process:

We use nx to run our e2e tests, and we use the distributed tasks and caching feature to speed up our test runs. The distributed tasks feature basically runs each test file as a distinct playwright run (you could just think of this as being exactly like playwright sharding, but sharded into each test file). The caching feature allows us to define all of the inputs for a task (files, env variables, external factors) and if the inputs match a previous run then the cached results are used (this saves us many minutes of CI run time!). We use the blob reporter to create a playwright blob report file as part of the outputs that are also stored in the cache after a run, or retrieved from the cache for a cached run. We then collect all of these blob reports in a subsequent ci step and merge them, publishing then as a consolidated html report. This is exactly the same as the recommended process for sharded playwright tests.

What we would like to do:

We would like to use the report portal reporter for the final step with the merged report.

A couple of acknowledgements:

• We know that report portal offers the ability to watch the results come through in a "live" fashion, and that this merged report is basically just sending through the launch results in the last step (after the fact).
• We also know that some of the results in the merged report are from cached results of previous runs.

I am getting an error when trying to use this as a reporter for the merge-reports feature:

processing test events
Error in reporter TypeError: Cannot read properties of undefined (reading 'name')
    at RPReporter.createSuites (/home/runner/work/experience-testing/experience-testing/node_modules/@reportportal/agent-js-playwright/src/reporter.ts:303:46)
    at RPReporter.onTestBegin (/home/runner/work/experience-testing/experience-testing/node_modules/@reportportal/agent-js-playwright/src/reporter.ts:364:40)
    at ReporterV2Wrapper.onTestBegin (/home/runner/work/experience-testing/experience-testing/node_modules/playwright/lib/reporters/reporterV2.js:55:157)
    at /home/runner/work/experience-testing/experience-testing/node_modules/playwright/lib/reporters/multiplexer.js:38:65
    at wrap (/home/runner/work/experience-testing/experience-testing/node_modules/playwright/lib/reporters/multiplexer.js:86:5)
    at Multiplexer.onTestBegin (/home/runner/work/experience-testing/experience-testing/node_modules/playwright/lib/reporters/multiplexer.js:38:45)
    at TeleReporterReceiver._onTestBegin (/home/runner/work/experience-testing/experience-testing/node_modules/playwright/lib/isomorphic/teleReceiver.js:140:157)
    at TeleReporterReceiver.dispatch (/home/runner/work/experience-testing/experience-testing/node_modules/playwright/lib/isomorphic/teleReceiver.js:63:12)
    at dispatchEvents (/home/runner/work/experience-testing/experience-testing/node_modules/playwright/lib/reporters/merge.js:53:22)
    at createMergedReport (/home/runner/work/experience-testing/experience-testing/node_modules/playwright/lib/reporters/merge.js:67:11)
    at mergeReports (/home/runner/work/experience-testing/experience-testing/node_modules/playwright/lib/cli.js:181:3)
    at t.<anonymous> (/home/runner/work/experience-testing/experience-testing/node_modules/playwright/lib/cli.js:95:7)

The line where this error is happening is here: https://github.com/reportportal/agent-js-playwright/blob/35fdbf1e80cea187bb930e2c2b46af967d89b855/src/reporter.ts#L303

If this is a bug that can be fixed, then that would be great!

If this is a more fundamental issue... then what is the issue? All I can think of is either:

1. A difference in the sequencing of calls when the reporter is called from tests vs the merge reports
2. The possibility that this reporter needs information that is only available during the actual test run and not included in the blob report.

[markwhitfeld]

A follow on question... Does the idea of replaying cached results from previous runs break the report portal paradigm?

• Let's say that the merge reports feature worked with this reporter, then the launch would include a repetition of results from a previous run. Both fresh and cached results would be part of the launch. Would this have any negative effects?
  • Potentially, if the testId is unique for a run, then the usage of past cached results could be detected by report portal, and handled accordingly.

An alternative is that I could just have report portal reporter configured during actual test runs.

• This would result in a launch only having a few tests and not representing the full test suite because the cached runs are not executed, and would therefore not report those tests.
• Is there potentially a way to somehow structure my launches so that this approach makes sense?

[DvaKompota]

... I am getting an error when trying to use this as a reporter for the merge-reports feature: ... If this is a bug that can be fixed, then that would be great!

If this is a more fundamental issue... then what is the issue? All I can think of is either:

1. A difference in the sequencing of calls when the reporter is called from tests vs the merge reports
2. The possibility that this reporter needs information that is only available during the actual test run and not included in the blob report.

We are having the exact same issue. Is there any existing or easily reachable solution to it @DzmitryHumianiuk? Also, it would be INCREDIBLY helpful to have an option to submit a report to ReportPortal only once at the end of a test run (not live on per-test basis) — we often have issues with RP server being overwhelmed with API calls from our CI, so reducing the number of requests from hundreds per run to just one would be a great help

[DzmitryHumianiuk]

@markwhitfeld, it looks like what you're describing is handled by our rerun function. You can check out the guide here: Rerun Developer's Guide.

I’m certain it’s implemented for Java, but I’m not sure if it’s available for the JavaScript client. @AmsterGet, can you confirm?

The idea is simple (https://reportportal.io/docs/developers-guides/RerunDevelopersGuide/#using-agent-1) — you can either use the rerun parameter to overwrite the results of the last launch with the same name:

{
  "name": "launch_name",
  "description": "some description",
  "mode": "DEFAULT",
ᐷ "rerun": true, 
}

Or you can use rerun + rerunOf to specify the exact launch UUID.

{
  "name": "launch_name",
  "description": "some description",
  "mode": "DEFAULT",
ᐷ "rerun": true,
ᐷ "rerunOf": "79446272-a439-45f9-8073-5ca7869f140b"
}

If you know for sure that multiple threads have been started and one of them created the launch in advance, you can pass its UUID to the other threads and try setting rerun=true.

Or just simply create a launch in advance with a separate call, and share the UUID with every other concurrent thread.

[DzmitryHumianiuk]

@DvaKompota,

I can break your question down into two parts:

1. Test sharding
2. Instance performance and post-import.

Regarding sharding, I’ve outlined existing options above that you can explore for launching tests. We'll also assess what needs to be done on the API side to further support this.

For performance, you should consider optimizing your instance. It might be worth upgrading to a more robust setup, as ReportPortal's capabilities scale proportionally with the hardware. ReportPortal can comfortably handle up to 5000 requests per second to its API without a drop in response time. You can check more here: Hardware Requirements.

Basic monitoring might help diagnose performance issues: Basic Monitoring Configuration.

As for post-import, yes, this feature is available. In version 24.1, we only supported loading ready-made jUnit report files. However, there’s great news — with version 24.2 Release v24.2, we’ve moved the Import functionality into a separate plugin.

This change allows anyone to write their own parser for any type of file or report, compile it into a jar file, and install it in ReportPortal via drag-and-drop, without needing to reinstall the app.

With this update, it's now possible to work with the community to create a custom parser for Playwright test reports (which should be within the capability of tools like GPT-4 or Claude 3.5).

So, feel free to contribute!