use playwright to do browser testing

[edewit]

in favour of #400

[vorburger]

merge as soon as I have to locally built & pass

Running 1 test using 1 worker
  1) [firefox] › demo-command.spec.js:3:1 › test blockly by creating command =======================

    browserType.launch:
    ╔══════════════════════════════════════════════════════╗
    ║ Host system is missing dependencies to run browsers. ║
    ║ Please install them with the following command:      ║
    ║                                                      ║
    ║     npx playwright install-deps                      ║
    ║                                                      ║
    ║ Alternatively, use apt:                              ║
    ║     apt-get install libgtk-3-0\                      ║
    ║         libasound2\                                  ║
    ║         libx11-6\                                    ║
    ║         libxcomposite1\                              ║
    ║         libxdamage1\                                 ║
    ║         libxext6\                                    ║
    ║         libxfixes3\                                  ║
    ║         libxrandr2\                                  ║
    ║         libxrender1\                                 ║
    ║         libxtst6\                                    ║
    ║         libpangocairo-1.0-0\                         ║
    ║         libpango-1.0-0\                              ║
    ║         libatk1.0-0\                                 ║
    ║         libcairo-gobject2\                           ║
    ║         libcairo2\                                   ║
    ║         libgdk-pixbuf-2.0-0\                         ║
    ║         libglib2.0-0\                                ║
    ║         libdbus-glib-1-2\                            ║
    ║         libdbus-1-3\                                 ║
    ║         libxcb-shm0\                                 ║
    ║         libx11-xcb1\                                 ║
    ║         libxcb1\                                     ║
    ║         libxcursor1\                                 ║
    ║         libxi6                                       ║
    ║                                                      ║
    ║ <3 Playwright Team                                   ║
    ╚══════════════════════════════════════════════════════╝

  1 failed
    [firefox] › demo-command.spec.js:3:1 › test blockly by creating command ========================

  Serving HTML report at http://localhost:9223. Press Ctrl+C to quit.
Failed to open browser on http://localhost:9223#?testId=93e88add21f52aab57b8-4d9c097d617b99892bf9

    1) 2 should create new command /demo
    2) 3 should execute /demo

  1 passing (51s)
  2 failing

  1) Storeys plugin test
       2 should create new command /demo:
     Error: Timeout of 50000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/project/storeys.test.js)
      at listOnTimeout (node:internal/timers:564:17)
      at process.processTimers (node:internal/timers:507:7)

  2) Storeys plugin test
       3 should execute /demo:
     Uncaught TypeError: Cannot read properties of undefined (reading 'value')
      at EventEmitter.<anonymous> (storeys.test.js:27:72)
      at EventEmitter.emit (node:events:525:35)
      at Client.<anonymous> (node_modules/mineflayer/lib/plugins/chat.js:120:9)
      at Client.emit (node:events:513:28)
      at FullPacketParser.<anonymous> (node_modules/minecraft-protocol/src/client.js:91:12)
      at FullPacketParser.emit (node:events:513:28)
      at addChunk (node_modules/readable-stream/lib/_stream_readable.js:298:12)
      at readableAddChunk (node_modules/readable-stream/lib/_stream_readable.js:280:11)
      at Readable.push (node_modules/readable-stream/lib/_stream_readable.js:241:10)
      at Transform.push (node_modules/readable-stream/lib/_stream_transform.js:139:32)
      at FullPacketParser._transform (node_modules/protodef/src/serializer.js:89:10)
      at Transform._read (node_modules/readable-stream/lib/_stream_transform.js:177:10)
      at Transform._write (node_modules/readable-stream/lib/_stream_transform.js:164:83)
      at doWrite (node_modules/readable-stream/lib/_stream_writable.js:409:139)
      at writeOrBuffer (node_modules/readable-stream/lib/_stream_writable.js:398:5)
      at Writable.write (node_modules/readable-stream/lib/_stream_writable.js:307:11)
(...)

[vorburger]

npx playwright install-deps

As discussed, I've added this in test-mineflayer/package.json, as npx playwright install --with-deps chrome.

That's better, but fails with ERROR: cannot install on debian distribution - only Ubuntu is supported.

I'll try to switch our build container from Debian to Ubuntu to see if that helps...

[vorburger]

I'll try to switch our build container from Debian to Ubuntu to see if that helps...

Done! See additional commit(s) added. With that, Playwright now successfully runs, and almost 😈 works.

As in, it worked once for me, but when I re-ran it a 2nd time, it fails like this, probably some race condition in the drag & drop:

Running 1 test using 1 worker
  1) [firefox] › demo-command.spec.js:3:1 › test blockly by creating command =======================

    locator.click: Error: strict mode violation: "text=When /" resolved to 2 elements:
        1) <text x="0" y="14" class="blocklyText">When /</text> aka playwright.$("text=When / >> nth=0")
        2) <text x="0" y="14" class="blocklyText">When /</text> aka playwright.$("text=When / >> nth=1")

    =========================== logs ===========================
    waiting for selector "text=When /"
      Error: strict mode violation: "text=When /" resolved to 2 elements:
        1) <text x="0" y="14" class="blocklyText">When /</text> aka playwright.$("text=When / >> nth=0")
        2) <text x="0" y="14" class="blocklyText">When /</text> aka playwright.$("text=When / >> nth=1")

    ============================================================

       8 |   await page.getByText("Events").click({ force: true });
       9 |   const whenEvent = page.getByText("When /");
    > 10 |   await whenEvent.click();
         |                   ^
      11 |
      12 |   await page.getByText("Actions").click({ force: true });
      13 |   const title = page.getByText("title").first();

        at /project/tests/demo-command.spec.js:10:19

  1 failed
    [firefox] › demo-command.spec.js:3:1 › test blockly by creating command ========================

  Serving HTML report at http://localhost:44427. Press Ctrl+C to quit.
Failed to open browser on http://localhost:44427#?testId=93e88add21f52aab57b8-4d9c097d617b99892bf9
    1) 2 should create new command /demo
    ✔ 3 should execute /demo (52ms)

  2 passing (51s)
  1 failing

Two more minor loose ends here which we should probably also address:

A. Downloading Firefox 105.0.1 (playwright build v1357)... Firefox 105.0.1 (playwright build v1357) downloaded to /root/.cache/ms-playwright/firefox-1357 I think we can probably save that by installing not just those missing dependencies but Firefox itself during the container build already as well. (I'll have a go at that in a minute.)

B. For me, when the Playwright tests fail, the entire ./test script never returns - because our docker run ... npm test never returns. I suspect that's because of this, perhaps you can figure out a way to also fix this:

  2 passing (51s)
  1 failing

  1) Storeys plugin test
       2 should create new command /demo:
     Error: Timeout of 50000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/project/storeys.test.js)
      at listOnTimeout (node:internal/timers:564:17)
      at process.processTimers (node:internal/timers:507:7)

[vorburger]

probably save that by installing not just those missing dependencies but Firefox itself during the container build already as well

Interestingly, even if we add firefox to the apt-get install in our Dockerfile-build, Playwright still re-downloads Firefox. Perhaps it wants to have a specific version, or likes to "control" it, and can't one pre-installed in the system? I've also noticed that it installs Firefox, AND Chromium, AND Webkit. Anyway. We can ignore that (for me), and merge this once the main problem above is resolved.

[edewit]

I know why it fails the second time because it already has the command setup as this is saved on the server in the config/storyes-web/new-scripts/ folder

[vorburger]

I know why it fails the second time because it already has the command setup as this is saved on the server in the config/storyes-web/new-scripts/ folder

Oh! That's why it worked for me the first time, right; thanks!!

Could/should the test should "clean up after itself" and remove the command after it's done?

Or (probably better actually?) should it delete its own old script BEFORE it starts?

Or I can also merge this as-is already, and we clean it up later... what do you prefer?

BUT there is now a merge conflict, which is just because I merged #426 in the meantime. Do you want to resolve it, or me?

PS: Is B. Playwright tests fail, the entire ./test script never returns super easy to fix, or do want to ignore that, for now? Fine with me also, just asking.

[edewit]

Let's merge it as it is, technically on a ci it's a clean environment each time, and if it would delete it before it starts it we should then also look that it might not be there and only clean when it is... it complicates the browser test a lot