Question: Using BackstopJS with Playwright: Unknown option --no-sandbox (webkit)

[mossmoss]

I asked this on Playwright's github, a contributor there advised not sending the --no-sandbox arg to webkit (I have tried that), and asking for help here. I realize this might not be a BackstopJS issue, but appreciate any help. Here's the issue I posted to Playwright:

We're using the latest (playwright:focal) Docker image of playwright (and have also tried with playwright:v1.8.0-focal) along with some code that tries to run BackstopJS with playwright:

https://github.com/zhumingcheng697/Backstop-Playwright

root@a8efc0637903:/regression-suite# npm list | grep -i playwright
+-- backstop-playwright@0.0.6
| +-- playwright@1.13.0 deduped
+-- playwright@1.13.0

We can run tests successfully with Chromium and Firefox but see the following when trying to test with webkit:

Type in a keyword to start: "test" (t), "approve" (a), "reference" (r)
t
No previous references exist for scenario 29 (Error).
Running REFERENCE for scenario 29 (Error).
Object-literal config detected.
COMMAND | Executing core for "reference"
  clean | backstop_data/bitmaps_reference/webkit/Error was cleaned.
createBitmaps | Selected 1 of 1 scenarios.
chooseBrowserType | Running in webkit.
          COMMAND | Command "reference" ended with an error after [1.157s]
          COMMAND | browserType.launch: Protocol error (Playwright.enable): Browser closed.
                    ==================== Browser output: ====================
                    <launching> /ms-playwright/webkit-1516/pw_run.sh --inspector-pipe --headless --no-startup-window --no-sandbox
                    <launched> pid=1138
                    [pid=1138][err] Cannot parse arguments: Unknown option --no-sandbox
                    [pid=1138] <process did exit: exitCode=1, signal=null>
                    [pid=1138] starting temporary directories cleanup
                    =========================== logs ===========================
                    <launching> /ms-playwright/webkit-1516/pw_run.sh --inspector-pipe --headless --no-startup-window --no-sandbox
                    <launched> pid=1138
                    [pid=1138][err] Cannot parse arguments: Unknown option --no-sandbox
                    [pid=1138] <process did exit: exitCode=1, signal=null>
                    [pid=1138] starting temporary directories cleanup
                    ============================================================
                        at processScenarioView (/regression-suite/node_modules/backstop-playwright/core/util/runPlaywright.js:64:77)
                        at module.exports (/regression-suite/node_modules/backstop-playwright/core/util/runPlaywright.js:37:10)
                        at /regression-suite/node_modules/p-map/index.js:46:16
REFERENCE failed for scenario 29 (Error).
Type in "auto run" at any time to start auto run.
Type in "approve all" at any time to approve all scenarios.
Type in "combine reports" at any time to combine all previously failed tests into one html report.
Type in "show list" to see a list of all scenarios.
Type in a valid index (0 to 35) or the scenario name to choose a scenario, type in "--" or "++" to choose the previous or the next scenario, if there is one, or type anything else or press enter to choose scenario 29 (Error) by default.

The code we're running with node looks like this at the start (I see the --no-sandbox option) but haven't been able to change it to something that works:

const fs = require("fs");
const YAML = require("yaml");
const readline = require("readline");
const backstop = require("backstop-playwright");
const combineReports = require("./combine-reports");

const { logStyle, resolveBrowserType } = require("./helper");

/**
 * Default Backstop configuration for all runs.
 *
 * @type {Object}
 */
const defaultConfig = {
    engine: "playwright",
    engineOptions: {
        "args": ["--no-sandbox"]
    },
    asyncCaptureLimit: 20,
    asyncCompareLimit: 100,
    debug: false,
    debugWindow: false
};

Removing this:

engineOptions: {
    "args": ["--no-sandbox"]
},

Results in:

Choose "chromium" (c), "firefox" (f), or "webkit" (w) as browser type?
w
/regression-suite/test.js:588
        if (!defaultConfig.engineOptions.browserType) {
                                         ^

TypeError: Cannot read property 'browserType' of undefined
    at Interface.<anonymous> (/regression-suite/test.js:588:42)
    at Interface.emit (events.js:375:28)
    at Interface._onLine (readline.js:364:10)
    at Interface._normalWrite (readline.js:509:12)
    at ReadStream.ondata (readline.js:216:10)
    at ReadStream.emit (events.js:375:28)
    at addChunk (internal/streams/readable.js:290:12)
    at readableAddChunk (internal/streams/readable.js:265:9)
    at ReadStream.Readable.push (internal/streams/readable.js:204:10)
    at TTY.onStreamRead (internal/stream_base_commons.js:188:23)

Using:

engineOptions: {
    "args": []
},

Results in:

Type in a keyword to start: "test" (t), "approve" (a), "reference" (r)
t
No previous references exist for scenario 29 (Error).
Running REFERENCE for scenario 29 (Error).
Object-literal config detected.
COMMAND | Executing core for "reference"
  clean | backstop_data/bitmaps_reference/webkit/Error was cleaned.
createBitmaps | Selected 1 of 1 scenarios.
chooseBrowserType | Running in webkit.
CREATING NEW REFERENCE FILE
CREATING NEW REFERENCE FILE
CREATING NEW REFERENCE FILE
CREATING NEW REFERENCE FILE
CREATING NEW REFERENCE FILE
Playwright encountered an error while running scenario "Error"
Error: page.goto: Error performing TLS handshake: The Diffie-Hellman prime sent by the server is not acceptable (not long enough).
=========================== logs ===========================
navigating to "http://www.exmaple.com/fake", waiting until "load"
============================================================
Playwright encountered an error while running scenario "Error"
Error: page.goto: Error performing TLS handshake: The Diffie-Hellman prime sent by the server is not acceptable (not long enough).
=========================== logs ===========================
navigating to "http://www.exmaple.com/fake", waiting until "load"
============================================================
Playwright encountered an error while running scenario "Error"
Error: page.goto: Error performing TLS handshake: The Diffie-Hellman prime sent by the server is not acceptable (not long enough).
=========================== logs ===========================
navigating to "http://www.exmaple.com/fake", waiting until "load"
============================================================
Playwright encountered an error while running scenario "Error"
Error: page.goto: Error performing TLS handshake: The Diffie-Hellman prime sent by the server is not acceptable (not long enough).
=========================== logs ===========================
navigating to "http://www.exmaple.com/fake", waiting until "load"
============================================================
Playwright encountered an error while running scenario "Error"
Error: page.goto: Error performing TLS handshake: The Diffie-Hellman prime sent by the server is not acceptable (not long enough).
=========================== logs ===========================
navigating to "http://www.exmaple.com/fake", waiting until "load"
============================================================

Run `$ backstop test` to generate diff report.

          COMMAND | Command "reference" ended with an error after [1.94s]
          COMMAND | [object Object]
REFERENCE failed for scenario 29 (Error).
Type in "auto run" at any time to start auto run.
Type in "approve all" at any time to approve all scenarios.
Type in "combine reports" at any time to combine all previously failed tests into one html report.
Type in "show list" to see a list of all scenarios.
Type in a valid index (0 to 35) or the scenario name to choose a scenario, type in "--" or "++" to choose the previous or the next scenario, if there is one, or type anything else or press enter to choose scenario 29 (Error) by default.

We're running Docker version 20.10.7, build f0df350 on CentOS Linux release 7.9.2009 (Core) - the Ubuntu version of the playwright image is Ubuntu 20.04.2 LTS.

[garris]

Hi @mossmoss -- first, good on you for having a cool github handle! Second double good on you for attempting to make playwright work with backstop! I would love to help but I have sooo little time. I will try to review this tomorrow. Feel free to bug me. Peace!

[garris]

I know I am super late to respond to this -- but I finally looked at these logouts. The first thing that came to mind is I wondered if just using playwright and safari (without backstop) works on Linux? I assume you already tried that and worked through the issue. Sorry I wasn't much help here.

[mossmoss]

Hi Garris, Thanks for circling back to this. I should have updated the issue. It turns out using a different version of playwright seems to have resolved the issue without having to change the --no-sandbox option - though my colleagues are more familiar with the issue. We switched from playwright focal to bionic. I'm not sure why it helped. Our Docker host is running on CentOS 7.9.2009 - which may be a reason changing playwright versions helped? Thanks again for your response!