phetsims / aqua

Automatic QUality Assurance
MIT License
2 stars 4 forks source link

Investigate tests that are not completing: #179

Closed zepumph closed 1 year ago

zepumph commented 1 year ago

Promoting from https://github.com/phetsims/special-ops/issues/234. There are some tests that are never being run. After https://github.com/phetsims/aqua/issues/178, this will greatly help us solve our issue here.

image UPDATE: Over the weekend there were many rows that said it tested "100%" of tests, but these are still blank, so this is probably the best list to investigate: image

zepumph commented 1 year ago

Looks like we are officially down to just the utterance-queue unit tests:

image

zepumph commented 1 year ago

This is occurring because headless chrome doesn't support speech synthesis. Locally by running aqua's grunt ct-node-client with this patch, the tests are passing:

Subject: [PATCH] d
---
Index: js/node-client/runTest.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/node-client/runTest.js b/js/node-client/runTest.js
--- a/js/node-client/runTest.js (revision dca9c2f71ccd4ec5907fd107994c04fc300d6679)
+++ b/js/node-client/runTest.js (date 1683740671172)
@@ -36,6 +36,18 @@
   const majorTimeout = 280000;
   const bailTimout = 400000;

+  testInfo = {
+    'snapshotName': 'snapshot-1683739496966',
+    'test':
+      [
+        'utterance-queue',
+        'top-level-unit-tests',
+        'unbuilt?ea'
+      ],
+    'timestamp': 1683740062842,
+    'url': 'qunit-test.html?url=..%2F..%2Fct-snapshots%2F1683739496966%2Futterance-queue%2Futterance-queue-tests.html%3Fea'
+  };
+
   const testInfoQueryParam = `testInfo=${encodeURIComponent( JSON.stringify( {
     test: testInfo.test,
     snapshotName: testInfo.snapshotName,

I added that patch to sparky, and then was editing up utterance-queue tests, commenting out sections to see what the trouble was, and found it!

setDefaultVoice is awaiting forever, but since voices never change, it can never set a default. So it waits forever. I'll work on a timeout for it in case we never change voices.

zepumph commented 1 year ago

I confirmed that this is now being tested. Closing.