Increase browser coverage on automated tests

[samreid]

From https://github.com/phetsims/chipper/issues/1273 and similar to https://github.com/phetsims/aqua/issues/80, it is now possible to run automated testing with more browser testing. Currently we are using the version of chromium that ships with puppeter, but puppeteer can also test in firefox now. Also, if we expand to playwright, it can additionally test on webkit.

Keep in mind this is separate from the problem of having separate dedicated computers running browser tests, which could be iPads, chromebooks, laptops, etc. This issue is about automated headless testing.

[zepumph]

Good progress here today! I was able to get a firefox instance loading with a tool in perennial called PERENNIAL/playwrightLoad. I'm not using this is CT just yet, but will soon!

[zepumph]

Alright, I loaded sparky ct clients up with 30 firefox and 70 puppeteer clients. Good luck us!

[zepumph]

I renamed the server processes to ct-browser-clients (instead of "chrome" specific).

[zepumph]

https://github.com/phetsims/perennial/commit/34c2a697222784b3bb97f12fc65b48d3457e02c6 was hiding a problem that launching a browser is timing out:

Worker307 Error from browser client: browserType.launch: Timeout 180000ms exceeded.
3|ct-brows | 2023-05-08T12:11:17:     at module.exports (/data/share/phet/continuous-testing/ct-browser-clients/perennial/js/common/browserPageLoad.js:57:55)
3|ct-brows | 2023-05-08T12:11:17:     at module.exports (/data/share/phet/continuous-testing/ct-browser-clients/perennial/js/common/playwrightLoad.js:28:10)
3|ct-brows | 2023-05-08T12:11:17:     at /data/share/phet/continuous-testing/ct-browser-clients/aqua/js/server/playwrightCTClient.js:31:23
3|ct-brows | 2023-05-08T12:11:17:     at Object.<anonymous> (/data/share/phet/continuous-testing/ct-browser-clients/aqua/js/server/playwrightCTClient.js:49:4) {
3|ct-brows | 2023-05-08T12:11:17:   name: 'TimeoutError'
3|ct-brows | 2023-05-08T12:11:17: }

I'm going to increase the timeout.

https://playwright.dev/docs/api/class-browsertype#browser-type-launch-option-timeout

[zepumph]

And after the above, we got this error:

Worker99 Error from browser client: browserType.launch:
3|ct-browser-clients  | 2023-05-08T12:23:02: ╔══════════════════════════════════════════════════════╗
3|ct-browser-clients  | 2023-05-08T12:23:02: ║ Host system is missing dependencies to run browsers. ║
3|ct-browser-clients  | 2023-05-08T12:23:02: ║ Missing libraries:                                   ║
3|ct-browser-clients  | 2023-05-08T12:23:02: ║     libdbus-glib-1.so.2                              ║
3|ct-browser-clients  | 2023-05-08T12:23:02: ╚══════════════════════════════════════════════════════╝
3|ct-browser-clients  | 2023-05-08T12:23:02:     at module.exports (/data/share/phet/continuous-testing/ct-browser-clients/perennial/js/common/browserPageLoad.js:57:55)
3|ct-browser-clients  | 2023-05-08T12:23:02:     at module.exports (/data/share/phet/continuous-testing/ct-browser-clients/perennial/js/common/playwrightLoad.js:28:10)
3|ct-browser-clients  | 2023-05-08T12:23:02:     at /data/share/phet/continuous-testing/ct-browser-clients/aqua/js/server/playwrightCTClient.js:31:23
3|ct-browser-clients  | 2023-05-08T12:23:02:     at Object.<anonymous> (/data/share/phet/continuous-testing/ct-browser-clients/aqua/js/server/playwrightCTClient.js:53:4) {
3|ct-browser-clients  | 2023-05-08T12:23:02:   name: 'Error'
3|ct-browser-clients  | 2023-05-08T12:23:02: }

@jonathanolson helped me install this dependency and we finally have our first CT reporting from Sparky Playwright Firefox.

I'll check back in later to see if we are having any issues from the playwright, vs the actual failing test.

@jonathanolson's commands to install a package that we were only medium confident that this would solve things: sudo dnf search dbus sudo dnf install dbus-glib.x86_64

[zepumph]

@jonathanolson and I discussed today, and would like to investigate a bit more into webkit in Playwright before closing this issue.

[zepumph]

• [x] Check the logs and see if firefox is causing any trouble.

[zepumph]

For testing webkit I was using this script:

( async () => {
  const playwrightLoad = require( './perennial/js/common/playwrightLoad.js' );
  const playwright = require( './perennial/node_modules/playwright' );

  await ( playwrightLoad( 'https://sparky.colorado.edu/continuous-testing/density/density_en.html?ea&brand=phet', {
    logConsoleOutput: true,
    testingBrowserCreator: playwright.webkit,
    logNavigation: true,
    logger: console.log,
    evaluate: () => {
      console.log( window.navigator.userAgent );
    }
  } ) );
} )();

When I try to run webkit on windows, I run into https://github.com/microsoft/playwright/issues/14105, and am trying to see if stubbing it is worth my time (it probably isn't, but here is a patch)

When I run this on sparky, I need these dependencies, perhaps @jonathanolson can help me with that sometime tomorrow:

[phet@sparky tmp]$ node js.js
browserType.launch:
╔══════════════════════════════════════════════════════╗
║ Host system is missing dependencies to run browsers. ║
║ Missing libraries:                                   ║
║     libicudata.so.66                                 ║
║     libicui18n.so.66                                 ║
║     libicuuc.so.66                                   ║
║     libwoff2dec.so.1.0.2                             ║
║     libharfbuzz-icu.so.0                             ║
║     libgstcodecparsers-1.0.so.0                      ║
║     libjpeg.so.8                                     ║
║     libwebp.so.6                                     ║
║     libenchant-2.so.2                                ║
║     libsecret-1.so.0                                 ║
║     libhyphen.so.0                                   ║
║     libXt.so.6                                       ║
║     libflite.so.1                                    ║
║     libflite_usenglish.so.1                          ║
║     libflite_cmu_grapheme_lang.so.1                  ║
║     libflite_cmu_grapheme_lex.so.1                   ║
║     libflite_cmu_indic_lang.so.1                     ║
║     libflite_cmu_indic_lex.so.1                      ║
║     libflite_cmulex.so.1                             ║
║     libflite_cmu_time_awb.so.1                       ║
║     libflite_cmu_us_awb.so.1                         ║
║     libflite_cmu_us_kal16.so.1                       ║
║     libflite_cmu_us_kal.so.1                         ║
║     libflite_cmu_us_rms.so.1                         ║
║     libflite_cmu_us_slt.so.1                         ║
║     libpcre.so.3                                     ║
║     libffi.so.7                                      ║
║     libGLESv2.so.2                                   ║
║     libx264.so                                       ║
╚══════════════════════════════════════════════════════╝
    at module.exports (/data/share/phet/continuous-testing/ct-main/perennial/js/common/browserPageLoad.js:57:55)
    at module.exports (/data/share/phet/continuous-testing/ct-main/perennial/js/common/playwrightLoad.js:28:10)
    at /data/share/phet/continuous-testing/ct-main/tmp/js.js:5:11
    at Object.<anonymous> (/data/share/phet/continuous-testing/ct-main/tmp/js.js:14:4) {
  name: 'Error'
}
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[zepumph]

Yesterday @jonathanolson and I spent some time seeing if these could get installed we got about half way and couldn't get any further. Then we found that RHEL isn't supported, https://github.com/microsoft/playwright/issues/6140.

So For safari we will take things up over in https://github.com/phetsims/special-ops/issues/243. Firefox is working well, and we will keep adapting the new client over in phetsims/aqua#178. Closing