Add the ability to control camera parameters

[jbphet]

In some of our test setups, we've been experiencing problems where the papers are not consistently detected and/or lose detection. It may be possible to improve this be adding the ability to control some of the camera parameters. @BLFiedler and I discussed this, and I'll do some initial investigation and prototyping.

[brettfiedler]

We found this reference as one way to possibly get at it simply: https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getSupportedConstraints

There are some parameters in OpenCV itself, but it's not clear if that's available in the web-browser (reference show Python and C++, no javascript access)

[brettfiedler]

List of parameters available for my setup (Logitech C920, win10):

Bolded the parameters I've found helpful for getting consistent program detection. Italicized parameters are the most helpful:

"aspectRatio": true, "autoGainControl": true, "brightness": true, "channelCount": true, "colorTemperature": true, (auto controlled by whiteBalanceMode) "contrast": true, "deviceId": true, "displaySurface": true, "echoCancellation": true, "exposureCompensation": true, "exposureMode": true, "exposureTime": true, "facingMode": true, "focusDistance": true, "focusMode": true, "frameRate": true, "groupId": true, "height": true, "iso": true, "latency": true, "noiseSuppression": true, "pan": true, "pointsOfInterest": true, "resizeMode": true, "sampleRate": true, "sampleSize": true, "saturation": true, "sharpness": true, "suppressLocalAudioPlayback": true, "tilt": true, "torch": true, "whiteBalanceMode": true, (controls colorTemperature) "width": true, "zoom": true

[jbphet]

I did some work on this, and found that there is a lot of variability in what the different cameras allow the code to control. For instance, my integrated web cam doesn't allow white balance control, but Brett's does. I was able to get some rudimentary things working, but the presence of the code seemed to mess up some of the cameras, which was the case for @terracoda. This all means that it will be fairly involved to create a UI to manipulate these parameters since the UI element will need to adapt to the particular camera in use. Since this effort was launched initially to help users - particularly @terracoda - get the papers to be recognized more consistently, we ended up trying some better web cams, and that is working well so far.

Because of the complexity of implementing this, coupled with the fact that we have a reasonable alternative for now, we are going to defer this effort and focus on other things. We will revisit it at some point and decide if we think it would be worth pursuing.

[jessegreenberg]

We are having trouble with detection of programs in codesign sessions so we would like to revisit this issue. @brettfiedler and I met to discuss next steps. We would like to:

• Unsupported constraints will crash when you try to set them. Lets handle these errors and also hopefully disallow setting of constraints that are not supported.
• Add the additional controls bolded/italicized in the above comments.
• Disable controls that the user's platform does not support.
• Add a reset button for defaults (mid range if there is no reasonable way to get defaults)
• Save settings to local storage for multiple sessions.
• Be on the alert for other changes in detection and framerate like we saw before. We will try to test on a number of platforms before finalizing.

[jessegreenberg]

I think this is ready for review and further testing @brettfiedler. It is now in the dev branch.

Here are the attributes you can control now:

• contrast
• exposureMode
• exposureTime
• focusDistance
• focusMode

The UI component for the constraint is only displayed if it is supported. If there is an error during change of a constraint, it should let the user know without crashing the Camera page. A reset button is at the bottom of the controls.

Some notes as we continue:

• I couldn't find reasonable defaults for start/reset and took a best guess. Midway through supported ranges looked really strange.
• The camera device seems to automatically save settings. I didnt save anything to localstorage but these values persist across sessions.
• Sharpness IS supported on my platform but it doesn't do anything.

[brettfiedler]

Two points to investigate:

1. Can we add a "Flip Camera Feed X-Axis" and "Flip Camera Feed Y-Axis" checkboxes in the camera sub-menu? I don't think this has anything to do with the cameras, but something we'd adjust in detectPrograms.js? So, independent of webcam. Still fits in the Camera sub-menu though for calibration purposes. Similar reasons to our inclusion in RaP CI: Hands and Quad CI: Objects.

2. @terracoda is on MacOS and Chrome using a Logitech c922. This is like my webcam, but better, so the camera itself should have the amenable parameters. However, she does not see any settings in the Camera sub-menu. Can we quickly investigate if this is a bug or MacOS getting in the way?

[brettfiedler]

During design meeting, we showed that the camera controls submenu only shows the controls for the first camera that loads up (based on system settings or history). Let's have it refresh when we change the camera.

[jessegreenberg]

Can we add a "Flip Camera Feed X-Axis" and "Flip Camera Feed Y-Axis"

I think I got this right. I tested by seeing that both the video feed displayed in the camera page and the detected points match up when the camera input is flipped.

during design meeting, we showed that the camera controls submenu only...

This should be fixed, I tested by swapping between usb and dedicated cameras on my laptop and seeing constrains apply to both as I expected.

@terracoda is on MacOS and Chrome using a Logitech c922. [...] Can we quickly investigate if this is a bug or MacOS getting in the way?

I am not sure if the changes for the other items will resolve this too. Its possible. These changes are still on the dev branch. @brettfiedler what else would you like to see before merging into main?

[jessegreenberg]

These controls did not work for recent interviews. During a discussion today the camera switching crashed for @brettfiedler. We are going to abandon this again for now. Ill move camera constraint controls off of the dev branch and into a feature branch. The camera flipping checkboxes ARE going to stay and get merged into main.

[jessegreenberg]

The controls for camera constraints have been removed from the dev branch. Now I am working on the 'flip' controls.

What is currently in the dev branch correctly flips camera and circle detection, but the color indices are all wrong and so the wrong programs are being detected when the camera is flipped. I might need to dig into the detection and color indices -> program number code a bit more to get this working.

[jessegreenberg]

Collecting some notes in this comment:

This patch works great for the "well detected" corners by reversing the order of color indices if there is a flip in one direction.

```patch Subject: [PATCH] Indicate that items have been sorted, see https://github.com/phetsims/scenery-phet/issues/815 --- Index: client/camera/detectPrograms.js IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/client/camera/detectPrograms.js b/client/camera/detectPrograms.js --- a/client/camera/detectPrograms.js (revision 582842952203647ffa97e1baf3b78bc0848ee719) +++ b/client/camera/detectPrograms.js (date 1705966376025) @@ -76,6 +76,26 @@ return colorsRGB.indexOf( colorDiff.closest( colorToRGB( matchColor ), colorsRGB ) ); } +/** + * Sort indexes to match the order of the code8400 array. My understanding is that keypoints are sorted by x position, + * assuming no camera flip. When we flip the camera in X or Y, the order of the keypoints is reversed. If we flip + * in both, the reverse cancels out. + * + * @param colorIndexes + * @param flipX + * @param flipY + */ +function sortIndexesForFlip( colorIndexes, flipX, flipY ) { + const indexes = colorIndexes.slice(); + + // This is like an XOR check for boolean + if ( flipX !== flipY ) { + indexes.reverse(); + } + + return indexes; +} + function shapeToId( colorIndexes ) { return code8400.indexOf( colorIndexes.join( '' ) ) % ( code8400.length / 4 ); } @@ -133,7 +153,7 @@ return false; } -function colorIndexesForShape( shape, keyPoints, videoMat, colorsRGB ) { +function colorIndexesForShape( shape, keyPoints, videoMat, colorsRGB, flipX, flipY ) { const shapeColors = shape.map( keyPointIndex => keyPointToAvgColor( keyPoints[ keyPointIndex ], videoMat ), colorsRGB @@ -147,7 +167,9 @@ remainingShapeColors.splice( closestColorIndex, 1 ); } ); - return shapeColors.map( color => colorIndexForColor( color, closestColors ) ); + const indexes = shapeColors.map( color => colorIndexForColor( color, closestColors ) ); + + return sortIndexesForFlip( indexes, flipX, flipY ); } export default function detectPrograms( { @@ -301,7 +323,7 @@ shape.reverse(); } - const colorIndexes = colorIndexesForShape( shape, keyPoints, videoMat, config.colorsRGB ); + const colorIndexes = colorIndexesForShape( shape, keyPoints, videoMat, config.colorsRGB, config.flipCameraFeedX, config.flipCameraFeedY ); const id = shapeToId( colorIndexes ); const cornerNum = shapeToCornerNum( colorIndexes ); ```

But it doesn't work for the "undefined" corners like the top right corner here:

When I flip the camera it really loses that corner:

For next time, lets see how the flip can be added around this part: https://github.com/phetsims/paper-land/blob/6e0564d2adf8ce298fc5d4508f9dd0731b971cc6/client/camera/detectPrograms.js#L390-L407

Some ChatGPT considerations: https://chat.openai.com/c/ae358083-b1c3-4938-8318-16ca9805e5ac

[jessegreenberg]

@brettfiedler mentioned that when flipping X and Y at the same time the detection worked as expected.

Since flipping in x/y individually is not trivial, one thing we might do is just have a "flip camera" option that just flips both. Assigning to @brettfiedler to comment on next steps for this issue.