Combine reports across tests

[aaronbauman]

Scenario: I'm running backstopjs tests in CircleCI environment, and I want to take advantage of parallelism by splitting up test config into multiple parallel runs. For example, each backstopjs run may contain just 1 or 2 scenarios, but I'm running 20 different scenarios at the same time.

Problem: Each test generates its own reporting, which is tedious to comb through.

Desired solution: For any given number of test runs, generate a single combined report, so that I can more easily assess visual regression results.

Is there an easy way to get at this?

[garris]

So, two things -- let me first address your parallel issue. BackstopJS already runs tests in parallel and combines results into one report. See https://github.com/garris/BackstopJS#capturing-screens-in-parallel

Second -- there is a whole other way of using BackstopJS where you run a backstop server and your test-runner (e.g. QUnit) implements a backstop helper to take screenshots in line with your other tests. This method allows you to asynchronously run an arbitrary number of backstop tests and aggregate these into one report -- this approach is not properly documented -- but you can follow along using documentation from the ember-backstop project

[aaronbauman]

Thanks - I'll check out the ember-backstop project.

I'm not sure how high I can ratchet backstop's concurrency within a single circle job before hitting resource limits, which is why i thought to run parallel circle instances.

[dannycalleri]

Hello! I think I run into the same problem when using ember-backstop along with ember exam and parallelization.

The command I'm using is: ember exam --split=12 --partition=1 --parallel=4 --load-balance

For some partitions, I get 3 or more folders inside the ember-backstop/backstop_data/bitmaps_test folder:

ember-backstop/backstop_data/bitmaps_test
├── 20230518-122916
│   ├── ...
│   └── report.json 👈
├── 20230518-122917
│   ├── ...
│   └── report.json 👈
└── 20230518-122919
    ├── ...
    └── report.json 👈

4 directories, 21 files

And I get one html report:

ember-backstop/backstop_data/html_report
├── ...
├── assets
│   └── fonts
│       ├── ...
├── ...
├── config.js 👈
├── diff.js
├── diverged.js
├── divergedWorker.js
├── index.html
└── index_bundle.js

3 directories, 14 files

The problem is that the config.js file inside the html_report will include the tests from just one of the above report.json files (usually the last one).

To get a report including all of them, I've to:

• join the ember-backstop/backstop_data/bitmaps_test/*/report.json files manually by concatenating the tests array inside each of them
• replace the tests array in the config.js file in ember-backstop/backstop_data/html_report with the concatenated result from the point above

Is there another supported way to do this? If the format of the array or the directory structure will change, my script will break, so I'd like to understand if there are alternative ways to do this.

Thanks for your help