testHash.testId cannot be overridden in the options, leading to multiple folders in bitmaps_test when running tests in parallel

Hello!

When running ember backstop-remote along with ember exam with the parallel option, I get multiple folders inside ember-backstop/backstop_data/bitmaps_test, each containing the snapshots taken by a particular browser instance, as visible below:

actual:

ember-backstop/backstop_data/bitmaps_test
├── 20230518-122916
│   ├── ...
│   └── report.json
├── 20230518-122917
│   ├── ...
│   └── report.json
└── 20230518-122919
    ├── ...
    └── report.json

expected:

ember-backstop/backstop_data/bitmaps_test
├── 20230518-122916
│   ├── ...
│   └── report.json

The command I'm using is: ember exam --split=12 --partition=1 --parallel=4 --load-balance. With this command I get at most 4 different folders.

The problem seems to be related to this row: https://github.com/garris/ember-backstop/blob/master/addon-test-support/backstop.js#L198

And the way the testRunTime is generated here: https://github.com/garris/ember-backstop/blob/master/addon-test-support/backstop.js#L70

Since that code will run once per each browser instance, each window will carry a different value for that variable. Thus, with 4 browser instances, I'll get 4 different values.

The goal is getting only one folder with one report.json file, as having more of them:

breaks the html_report - that for instance, will show only some of the failing tests
breaks the approval process as well, as visible in the code here: https://github.com/garris/BackstopJS/blob/master/core/command/approve.js#L8

The proposed solution would be passing a new testId option here: https://github.com/garris/ember-backstop/blob/master/addon-test-support/backstop.js#L234

When available, it will take precedence over the one generated on the window.

I just created a Pull Request that solves the problem here: https://github.com/garris/ember-backstop/pull/83

What do you think? Thanks 🙏

garris / ember-backstop

testHash.testId cannot be overridden in the options, leading to multiple folders in bitmaps_test when running tests in parallel #82