Investigate MechaMarkers integration

[zepumph]

We are beginning to investigate using mechamarkers, a creation from partners in the Atlas lab, as an input modality to phetsims. This morning, developers from that project (@Petroochio and @clementzheng) met with me and @jessegreenberg to discuss the implementations of the two projects as they pertain to creating an adapter to join the two.

We were able to create a hello world example that used the 18th Aruco marker to trigger a reset of Gravity Force Lab when present on the screen.

In general here is how the stack works:

• Open https://docs.google.com/document/d/1ce4HulQv9kNQk0rpKPbm8Y4u21zj5EKkpUOz5s34xuY/edit and find mechamarkers application for mac/windows and run it.
• get the 18th aruco marker that you can display to the camera, and in the app make sure that it is being recognized.
• Open up https://glitch.com/edit/#!/reset-gravity-force-lab and run it. When the marker is shown, then the sim will reset.

Under the hood on the phet side, we just added a simple PostMessage line directly to ResetAllButton. This doesn't feel like something that we are going to be able to maintain, but it did make for a nice "hello world"

From here I think that @jessegreenberg and I should investigate the most reasonable places to add adapter code to. I will try to discuss tomorrow during a11y dev meeting.

[zepumph]

Here is the scrap from ResetAllButton:

Index: js/buttons/ResetAllButton.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- js/buttons/ResetAllButton.js    (revision 5a7ae48875064e372f1107a4c27b75dadcefafa1)
+++ js/buttons/ResetAllButton.js    (date 1581373842971)
@@ -77,6 +77,19 @@

     ResetButton.call( this, options );

+    window.addEventListener( 'message', event => {
+      try {
+        const json = JSON.parse( event.data );
+
+        if ( json.type === 'RESET' ) {
+          options.listener();
+        }
+      }
+      catch( e ) {
+        //
+      }
+    } );
+
     // @private - Mirrored property of `buttonModel.isFiringProperty`, but is phet-io instrumented.
     this.isFiringProperty = new DerivedProperty( [ this.buttonModel.isFiringProperty ], function( a ) { return a; }, {
       tandem: options.tandem.createTandem( 'isFiringProperty' ),

Here is a list of potential ways forward some may be connected but others not really:

• Branch scenery to add "low level" input channels for mechamarkers to use
• Branch GFL and just add specific hooks to the components needed for just this sim while prototyping.
• Create a new repo with adapter code for mechamarkers to work with phetsims.
• Attach to listeners that already exist, like PressListener/DragListener/KeyboardDragListener
• Create a new scenery listener like MarkersListener which is set up to listen to these changes.

[zepumph]

After discussing with @jessegreenberg today, we think that the best way forward for prototyping is to create a sim-specific module that is passed the model. Since mechamarkers uses a webSocket, we are very flexible as to how we add it into our unbuilt code. For now we will keep it to one sim.

We can keep it on master with a line like: assert && phet.queryParameters.markerInput && MarkerInput.go( model );

But first I will want to make sure that we aren't including MarkerInput in the build files.

[zepumph]

We have given up on mechamarkers. This should be cleaned up and related issues closed. We are now using MediaPipe.

[zepumph]

I can cleanup mechamarkers on main and close.

[zepumph]

We are going to use MediaPipe from here.