phetsims / aqua

Automatic QUality Assurance
MIT License
2 stars 4 forks source link

Can CT fuzzing report which screen was active when a fuzz error occurs? #142

Open samreid opened 2 years ago

samreid commented 2 years ago

For sims with many different kinds of screens, it would be very useful for CT to report which screen had the failure. Would this be easy/difficult? Would there be false positives?

Assigning to @jonathanolson for initial discussion.

zepumph commented 2 years ago

This didn't work as I expected it to, so I'll have to do some more research:


Index: friction/js/friction/model/FrictionModel.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/friction/js/friction/model/FrictionModel.js b/friction/js/friction/model/FrictionModel.js
--- a/friction/js/friction/model/FrictionModel.js   (revision 8c86e9c90e5452d758ac634bcb0239b91b356e76)
+++ b/friction/js/friction/model/FrictionModel.js   (date 1654795154388)
@@ -10,9 +10,9 @@

 import BooleanProperty from '../../../../axon/js/BooleanProperty.js';
 import DerivedProperty from '../../../../axon/js/DerivedProperty.js';
-import Property from '../../../../axon/js/Property.js';
 import Emitter from '../../../../axon/js/Emitter.js';
 import NumberProperty from '../../../../axon/js/NumberProperty.js';
+import Property from '../../../../axon/js/Property.js';
 import Bounds2 from '../../../../dot/js/Bounds2.js';
 import dotRandom from '../../../../dot/js/dotRandom.js';
 import Range from '../../../../dot/js/Range.js';
@@ -274,6 +274,9 @@
       this.numberOfAtomsShearedOffProperty.value += 1;
     } );

+    setTimeout( () => {
+      throw new Error( 'oh boy' );
+    } );
     // @public (read-only)
     // {boolean} - has the atom been "successfully" interacted with. This subjective term is defined based on the
     // pedagogical goals of the sim (to rub the other book)
Index: chipper/js/initialize-globals.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/chipper/js/initialize-globals.js b/chipper/js/initialize-globals.js
--- a/chipper/js/initialize-globals.js  (revision 331185eb7c41175361cc23ee0effeb59e4ed86a1)
+++ b/chipper/js/initialize-globals.js  (date 1654795103469)
@@ -890,7 +890,13 @@
         if ( a && a.error && a.error.stack ) {
           stack = a.error.stack;
         }
+        let screenNumber = -1;
+        if ( _.hasIn( window, 'phet.joist.sim' ) ) {
+          screenNumber = phet.joist.sim.screens.indexOf( phet.joist.sim.screenProperty.value );
+        }
+        debugger;
         phet.chipper.reportContinuousTestResult( {
+          screenNumber: screenNumber,
           type: 'continuous-test-error',
           message: message,
           stack: stack