Make a Node-based CT client

zepumph commented 1 year ago

Instead of relying on continuous-loop, we can set up a puppeteer client and have much more control over rerunning tests, getting the console output, and handling errors. @jonathanolson will take the lead, and this is coming from https://github.com/phetsims/special-ops/issues/234 and https://github.com/phetsims/aqua/issues/173

jonathanolson commented 1 year ago

Initial commit in https://github.com/phetsims/aqua/commit/f4b952e59121ea2ff2c654f58ea10b8d51a3697e

zepumph commented 1 year ago

Taking a look at this now and running some local testing with it.

zepumph commented 1 year ago

This is looking great! Thanks @jonathanolson.

[x] we fixed a bug above where playwright support meant that we lost the "this" of the page.
[x] We need to support id checking
[x] It would be nice to get some of these onto CT to see how they go.
[x] Perhaps this would be a nice time to use pm2 cluster mode where we can just have 50 of these all running a forever loop wrapping runNextTest. Though it may clog up the pm2 list to an unacceptable level.

zepumph commented 1 year ago

I should test runNextTest with some options like id and browserCreator (for firefox).

zepumph commented 1 year ago

Things seem to be running well on sparky. I supported cluster mode, and it is working seemlessly from what I can tell:

I don't know if we want to have 100 of these though. It seems a bit messy. @jonathanolson thoughts?

zepumph commented 1 year ago

[ ] One error I'm seeing out of testing on sparky:

 You have triggered an unhandledRejection, you may have forgotten to catch a Promise rejection:
5|ct-node- | Error: Did not get next-test message in 400000ms
5|ct-node- |     at /data/share/phet/continuous-testing/ct-node-client/aqua/js/node-client/runTest.js:146:19

zepumph commented 1 year ago

Here is what one CT error looks like from this new client:

molarity : interactive-description-fuzzBoard : unbuilt
http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/molarity/molarity_en.html?continuousTest=%7B%22test%22%3A%5B%22molarity%22%2C%22interactive-description-fuzzBoard%22%2C%22unbuilt%22%5D%2C%22snapshotName%22%3A%22snapshot-1683744638744%22%2C%22timestamp%22%3A1683745232137%7D&brand=phet&ea&fuzzBoard&supportsInteractiveDescription=true
Query: brand=phet&ea&fuzzBoard&supportsInteractiveDescription=true
Uncaught Error: Assertion failed: no item found that has focus
Error: Assertion failed: no item found that has focus
at window.assertions.assertFunction (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/assert/js/assert.js:28:13)
at assert (ComboBoxListBox.ts:311:14)
at getFocusedItemNode (ComboBoxListBox.ts:238:75)
at inputEvent (Input.ts:1899:69)
at dispatchToListeners (Input.ts:1939:11)
at dispatchToTargets (Input.ts:1851:9)
at dispatchEvent (Input.ts:1119:11)
at dispatchPDOMEvent (Input.ts:800:20)
at apply (PhetioAction.ts:158:16)
at execute (Input.ts:1597:23)
[URL] http://127.0.0.1/continuous-testing/aqua/html/sim-test.html?url=..%2F..%2Fct-snapshots%2F1683744638744%2Fmolarity%2Fmolarity_en.html&simQueryParameters=brand%3Dphet%26ea%26fuzzBoard%26supportsInteractiveDescription%3Dtrue&duration=40000&testInfo=%7B%22test%22%3A%5B%22molarity%22%2C%22interactive-description-fuzzBoard%22%2C%22unbuilt%22%5D%2C%22snapshotName%22%3A%22snapshot-1683744638744%22%2C%22timestamp%22%3A1683745232137%7D
[NAVIGATED] http://127.0.0.1/continuous-testing/aqua/html/sim-test.html?url=..%2F..%2Fct-snapshots%2F1683744638744%2Fmolarity%2Fmolarity_en.html&simQueryParameters=brand%3Dphet%26ea%26fuzzBoard%26supportsInteractiveDescription%3Dtrue&duration=40000&testInfo=%7B%22test%22%3A%5B%22molarity%22%2C%22interactive-description-fuzzBoard%22%2C%22unbuilt%22%5D%2C%22snapshotName%22%3A%22snapshot-1683744638744%22%2C%22timestamp%22%3A1683745232137%7D
[NAVIGATED] about:blank
[NAVIGATED] http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/molarity/molarity_en.html?continuousTest=%7B%22test%22%3A%5B%22molarity%22%2C%22interactive-description-fuzzBoard%22%2C%22unbuilt%22%5D%2C%22snapshotName%22%3A%22snapshot-1683744638744%22%2C%22timestamp%22%3A1683745232137%7D&brand=phet&ea&fuzzBoard&supportsInteractiveDescription=true
[CONSOLE] enabling assert
[CONSOLE] continuous-test-load
[CONSOLE] Assertion failed: no item found that has focus
[PAGE ERROR] Error: Error: Assertion failed: no item found that has focus
at window.assertions.assertFunction (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/assert/js/assert.js:28:13)
at ComboBoxListBox.getFocusedItemNode (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/sun/js/ComboBoxListBox.js:251:15)
at Object.keydown (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/sun/js/ComboBoxListBox.js:185:75)
at Input.dispatchToListeners (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/scenery/js/input/Input.js:1714:25)
at Input.dispatchToTargets (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/scenery/js/input/Input.js:1743:12)
at Input.dispatchEvent (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/scenery/js/input/Input.js:1674:10)
at Input.dispatchPDOMEvent (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/scenery/js/input/Input.js:998:12)
at Input.keydownAction.PhetioAction.phetioPlayback (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/scenery/js/input/Input.js:708:21)
at PhetioAction.execute (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/tandem/js/PhetioAction.js:127:17)
at Input.keyDown (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/scenery/js/input/Input.js:1449:24)
[CONSOLE] continuous-test-error

id: "Sparky Node Puppeteer"
Snapshot from 5/10/2023, 12:50:38 PM

zepumph commented 1 year ago

[ ] launchArgs need to be moved over for the puppeteer node client: https://github.com/phetsims/aqua/blob/146014f7c6c3ff30a88e94721872e2bf7f3105a3/js/server/puppeteerCTClient.js#L39-L54

zepumph commented 1 year ago

We want to have this new client be the way of the future, let's phase out the old pattern

zepumph commented 1 year ago

Oops, I forgot about all the great puppeteer options we still need.

zepumph commented 1 year ago

Ok, we now have:

bayes: 16 node puppeteer clients

sparky: 5 browser firefox, 5 browser puppeteer 65 node puppeteer 25 node firefox.

I'll check in tomorrow.

zepumph commented 1 year ago

[x] Check the logs and see if firefox is causing any trouble.

zepumph commented 1 year ago

It has been helpful to have objects like this for testing:

  // testInfo = {
  //   'snapshotName': 'snapshot-1684173607587',
  //   'test': [ 'geometric-optics', 'phet-io-wrappers-tests', 'assert' ],
  //   'url': 'qunit-test.html?url=..%2F..%2Fct-snapshots%2F1684173607587%2Fphet-io-wrappers%2Fphet-io-wrappers-tests.html%3Fsim%3Dgeometric-optics%26phetioDebug%3Dtrue%26phetioWrapperDebug%3Dtrue',
  //   'timestamp': 1684172635047
  // };
  testInfo = {
    'snapshotName': 'snapshot-1684173607587',
    'test': [ 'geometric-optics', 'phet-io-wrappers-tests', 'assert' ],
    'url': '../../ct-snapshots/1684173607587/phet-io-wrappers/phet-io-wrappers-tests.html?sim=geometric-optics&phetioDebug=true&phetioWrapperDebug=true',
    'timestamp': 1684172635047
  };

zepumph commented 1 year ago

Ok. I'm feeling good about this issue. Over the weekend tests were running very well, save a couple of small problems.

I found that CT snapshots weren't getting transpiled and so the phet-io-wrappers tests were breaking, but https://github.com/phetsims/aqua/commit/e143df1c96035160fb3d1e6ea2bdb95faf4c7fec solved that.

@jonathanolson and I also updated some of the criteria for keeping snapshots, keeping the last 20 instead of 3 now, so that we can reproduce/test problems easier. Space isn't an issue on Sparky for that, so we aren't worried.

I think we are ready to close this issue. The rest will be handled by side issues. Thanks @jonathanolson for getting this to such a nice spot.

phet-dev commented 10 months ago

Reopening because there is a TODO marked for this issue.

phetsims / aqua

Make a Node-based CT client #178