Artwork needed for Camera Input: Hands

[terracoda]

Over in #491, we are working out the words to describe how customize the hand movement.

Here's how I currently have it described

Customize hand movement. [ ] Flip camera view of hands between frontal and horizontal views [ ] Flip hand operation between mirrored hand control and opposite hand control

Here are some sample hand drawings Sample hands from https://all-free-download.com/free-vector/download/free-vector-hands-human-hands_38058.html

Then connect a natural looking hand with the sim's hands... (working on an example)

[terracoda]

These are not very good. Just tryin to get an idea across for frontal camera view with the hands Directly mirrored, or

Opposite hand control

[terracoda]

@emily-phet, could you elaborate on what you have in mind for a graphic?

[terracoda]

Maybe we need a sim screen in front of the human hands?

[zepumph]

This is blocking publication of the sim. What are the goals here?

[terracoda]

Personally, I don't think this should block publication as long as the text is understandable.

[terracoda]

Here's a rough idea with a screen in front of the hands.

[zepumph]

It isn't clear to me what artwork you are interested it. What size generally do you want this? Is it an icon? Something for the options dialog as well as the website if we were going to filter on it/put it next to sims that supported it? Perhaps the "screen" portion of it should be general and not sim-specific. Like two hands and a blurry screen background or something. If you were trying to have an image showing how RAP works in the options Dialog, then my ideas are not the direction to go.

[terracoda]

I hadn't considered a blurry screen. That's an interesting idea.

And actually, it needs to be very simple - iconic in nature and ideally black and white. I was just using the image from the home screen button because I can't seem to draw.

Ideally, a screen on an angle with hands in front would be good.

[terracoda]

Is this better @zepumph

Camera Input: Hands With "cameraInput=hands" use your hands to move the sim's Left and Right Hands.

Customize hand movement [ ] Flip camera view of hands between frontal and horizontal views

[ ] Flip hand operation between mirrored hand control and opposite hand control

[terracoda]

I think we may need better images of the hands, but I think we are getting closer to simple and iconic.

[terracoda]

Another options for

[ ] Flip camera view of hands between frontal and horizontal views

[terracoda]

@emily-phet and @zepumph, assigning to you both for comment.

[terracoda]

I like the original order now, too. Mirrored ot not mirrored - frontal vertical view or over-the-top horizontal view.

Camera Input: Hands With "cameraInput=hands" use your hands to move the sim's Left and Right Hands.

Customize hand movement [ ] Flip hand operation between mirrored hand control and opposite hand control

[ ] Flip camera view of hands between frontal and horizontal views

[emily-phet]

@terracoda Did you reach out to @Ashton-Morris about this? If not, I can ask him if he could put something together quickly for this when I have some afternoon meetings with him. JB is out so we'll have time for that.

[Ashton-Morris]

I am going to brainstorm and try to come up with some ideas.

[terracoda]

Thanks @Ashton-Morris, I forgot to reach out to you.

[Ashton-Morris]

I have tried with some various perspectives. What do you all think?

[terracoda]

These look way more interesting than the ones I hacked together.

I personally like the straight-on perspective as I can see the letters there the best.

[emily-phet]

I like the straight on perspective, or the upper left one in your comment above. Let's have the sim hands look like the hands in the sim RAP, and distinguish them from the representation of the real human hands. Also - if the dimensions of the screen could be narrower, that would be great. Right now it looks more like a TV than a computer screen to me.

[terracoda]

The one in the upper left was my second favourite

[Ashton-Morris]

Update: I have made it more like a monitor a bit. Also added the Sim hands.

Should the sim hands be intentionally red still or do you all think black and white only would be best?

[zepumph]

Since we have such good progress here, I'm going to mark this as blocking to keep the conversation going.

In terms of adding this to the controls themselves:

• We could make this a radio button group where the icons are the UI.
• We could add like a "double arrow" swap to the icon and keep the flipping checkbox
• We could highlight the "selected" icon when checked, and grey out the other.

@terracoda what are you thinking about the above icons?

@terracoda what are you thinking about the final form of the preference.

[zepumph]

Next steps here from a conversation with @emily-phet:

1. We like a "toggle" style icon UI, something like Color Vision or Balancing Act:

 

3. Create heading for the UI. Something like "Flip hands up/down" and "Flip hands left/right"
4. Do we need additional onscreen text in addition? @emily-phet and I aren't quite sure. This is a question for @terracoda, if so, then we need to figure out how to attach that to the heading/icon combo.
5. Solidify the images. I think that we should work with @Ashton-Morris to solidify the design and then we can give the idea to @mariahmaephet.

[terracoda]

Here's the current text set-up in the Preferences Menu for Reference

Regarding the suggested heading, I am not understanding "Flip hands up/down" and "Flip hands left/right".

[terracoda]

We need to communicate two things visually

1. Your camera can be in front of you, or overhead
2. You can flip the hands

I think we are currently missing something. Are people expected to flip the hands if they are using an over the head camera?

I am wondering if all the information is visually captured yet. I am putting the text and potential visual together to assess what we need to represent.

[terracoda]

Please help me understand by comparing these two mock-ups Current Text Only Representation

Very rough idea of a potential visual representation.

How do we or do we need to represent the Camera View with the Visual Icon Representation?

[terracoda]

In the visual representation above, there are two separate choices to be made - one for the camera and one for your hands.

[emily-phet]

@terracoda Let's talk about this briefly you and I on Tuesday. I think that will be simplest. No need to look into it further before we talk.

[emily-phet]

Discussed with @terracoda and @jessegreenberg on 10/11/22. We concluded that graphic representation is not necessary (simplifying some of this issue).

@terracoda is going to return to working on language for on-screen for two (total) checkboxes. I suggested (after discussion with all and input from @jessegreenberg) something like: "Mirror horizontal movement" and "Mirror vertical movement".

We also concluded short, and without reference to any specific scenario would be the most generalizable and clear, given we don't know the camera setups people will use.

[zepumph]

Just a ping here. We are getting close to all other issues finishing up, and this is still marked as blocking RAP. Please change the blocking label or provide the wording updates desired. Thanks!

[zepumph]

@emily-phet and I feel like this is fine to go to RC with, but we will want new wording before we publish.

[terracoda]

@emily-phet, I am still a little fuzzy on how I can describe the set-up with one set of radio buttons. I think I need to assume vertical movement happens with a straight-on camera and horizontal movement happens with an over-the-head camera. That's not a choice, but an assumption.

Then the user can choose to change the hand association from a directly mirrored mapping to an opposite mapping. Now, how to describe that?

Does 4 radio buttons make sense to you @emily?

Customize hand movement Choose vertical or horizontal hand movements to control the sim. [x] Mirrored vertical movement [ ] Mirrored horizontal movement [ ] Opposite vertical movement [ ] Opposite horizontal movement

Note "mirrored vertical movement" is the default selection.

[terracoda]

Only one radio button can be selected.

[terracoda]

I am not sure "Opposite" is the correct term. "Natural" might be better.

Oh, I just realized I have this all backwards. The hands in the sim face the same direction as me. The left hand in the sim is controlled naturally by my own left hand.

Not sure why this was so confusing for me to understand.

Customize hand movement Choose vertical or horizontal hand movements to control the sim. [x] Natural vertical movement [ ] Mirrored vertical movement [ ] Natural horizontal movement [ ] Mirrored horizontal movement

[emily-phet]

I think something simple like: [ ] Flipped vertical movement [ ] Flipped horizontal movement

would be sufficient. To me, it's ok if you wouldn't necessarily know which specific direction your choosing ("natural" or "mirrored" from the menu. You would be seeking this option if you need to have it flipped. So in that situation, you would know. Otherwise, you might be curious what happens when you flip it...so again, you would know. I do like your "natural vs mirrored", but I think four radio buttons will feel very heavy for this option. If you need this, you know you need it changed. To me that seems more important than a person needing to identify specifically if they want "natural" or "mirrored". I like the approach of emphasizing the change (e.g., using the word "flip" emphasizes the change in state) rather than the current state.

When Jesse, you, and I met about this awhile back, Jesse was able to find some other examples using this same kind of emphasis on the change rather than the state. That doesn't make it the best solution, but at least it's not completely uncommon. :)

[terracoda]

I almost posted "flipped" with only 2 options with only 2 radio buttons, but I was thinking that there would be no option for "natural horizontal movement", but now I realize that natural horizontal movement is just the default if the learner is using an over-the-head camera.

If we only have 2 radio buttons as in the following design: Customize hand movement Flip hand operation to support vertical or horizontal hand movements to control the sim. [ ] Flipped vertical movement [ ] Flipped horizontal movement

We still have a problem. How does a user get back to "natural movement" if they choose to flip the hands?

As you said, we have to assume the user (teacher or learner) has a physical set-up, either a face-to-face camera or an over-the-head camera, or even a camera directly underneath. They may need to flip the hands to optimize the situation. If they change their physical set-up, they may need to un-flip the hands or put them back to a natural orientation.

With 2 radio buttons I think the interaction has to juxtapose natural versus mirrored. The user and their camera determine whether they are doing vertical or horizontal movement.

Customize hand movement If needed, flip the hands that control the sim to support different camera setups. [x] Natural hand control with vertical or horizontal movement. [ ] Flipped hand control with vertical or horizontal movement.

OR MAYBE something more explicit? Customize hand movement If needed, flip the hands that control the sim to support different camera setups. [x] Natural hand control. Best with face-to-face camera using vertical hand movements. [ ] Flipped hand control. Best with over-the-head camera using horizontal hands movement.

Possible Voicing Design: Focus on help text reading block, voice: "Customize hand movement. If needed, flip the hands that control the sim to support different camera setups."

Focus on toggle Interaction, voice: "Natural hand control. Best with face-to-face camera using vertical hand movements." Up toggling to flipped hand control, voice: "Flipped hand control. Best with over-the-head camera using horizontal hands movement."

Possible Toggle Interactions

I hope this all makes sense.

[emily-phet]

Ahh - I didn't catch this until just now. I think we need only two checkboxes, not radio buttons.

Horizontal and vertical movement is independent. My recommendation is to use the text I suggested above (using "flipped"), and for each of the two options there is a checkbox. The default is unchecked for both.

Another option would be to have to sets of two radio buttons:

• Vertical option radio buttons: One radio button is for the "natural" state, the other radio button is for the "mirrored" state.
• Horizontal option radio buttons: One radio button for the "natural" state, the other radio button for the "mirrored" state.

If @terracoda is on board with taking the approach of emphasizing the change in state, rather than describing the two different states possible for vertical and horizontal, then the two flipped checkboxes is the simplest route, and the one I recommend.

[terracoda]

I'll get back to this later this evening. I think we are finally getting on the same page.

[brettfiedler]

(not intended to derail current path here) Popping into the issue just to note that (at least when I requested the ability to flip the camera feed), I was envisioning it much the way game developers implement control customization, in which case they use the word "inversion" or "inverting controls", "Invert x- or y- axis", etc.

In case that phrase is helpful, it is very common.

[terracoda]

OK, I prefer to keep it simple, too. Let's stick with checkboxes. I think this is what Emily is suggesting.

The words are still a bit confusing me. Am I supposed to describe an interaction that is customizing how the hands in the sim move, or how hand input operates?

Customize hand movement or Customize hand operation If needed, flip hand [[movement/operation]] to support different camera setups. [ ] Flip vertical hand movement. [ ] Flip horizontal movement.

Now thinking about how this would sound with a screen reader or with Voicing.

[terracoda]

Voicing Design

Move focus to reading block:

Customize hand movement. If needed, flip hand [[movement]] to support different camera setups.

Checking a checkbox with Voicing Move focus to first checkbox:

Flip vertical hand movement.

Toggle checkbox with Space and here the context response:

Your right hand moves up and the sim's right hand moves down.

Toggle checkbox again to unchecked with Space and here the context response:

Your right hand moves down and the sim's right hand moves down.

Move focus to second checkbox:

Flip horizontal hand movement.

Toggle checkbox with Space and here the context response:

Your right hand moves up and the sim's right hand moves down.

Toggle checkbox again to unchecked with Space and here the context response:

Your right hand moves down and the sim's right hand moves down.

I don't think this is working. Met with @BLFiedler

[brettfiedler]

@terracoda and I met briefly to discuss this. Based on the implementation in the sim, which functionally is just flipping the X or Y axis of the camera, I suggested this be taken more from a troubleshooting perspective to address what each checkbox will do for your given camera orientation. Goals for the below is to be very explicit about exactly what each checkbox does, as well as the contexts you might use them, as well as providing more detail about the camera-sim connection. Proposed something similar to the below text:

---

Camera Input: Hands Troubleshooting

The simulation assumes your camera is facing you as with a traditional laptop webcam. If needed, use the options below to change the camera detection of your hands to match your hand movements.

Are your hands mapped to the wrong hands in the sim? [ ] Flip camera Y-axis.

Are your hands going in the wrong direction in the sim? [ ] Flip camera X-axis.

[terracoda]

Scenarios we are troubling shooting.

Default scenario (nothing needs to be checked) User has a face-to-face camera and their own hands operate the same hands in the sim, and hand direction is mapped identically in the sim.

When the physical camera is turned on the Y-axis... ...the camera detection also needs to be rotated on the Y-axis (see incorrect hand mapping in video screen). To fix this issue the user likely needs to check "Flip camera Y-axis."

When the physical camera is turned on the X-axis (e.g. over-head)... ...the camera detection also needs to be rotated on the X-axis (see incorrect hand mapping in video screen). To fix this issue the user likely needs to check "Flip camera X-axis."

[terracoda]

Troubleshooting Camera Input: Hands

The simulation assumes your camera is facing you as with a traditional laptop webcam. If this is your setup, you are ready to go.

If your camera is behind you, physically rotated on the Y-axis, your hand movements may feel like they map to the wrong objects in the sim. Try flipping the Y-axis. [ ] Flip camera Y-axis.

If your camera is over head, physically rotated on the X-axis, your hand movements may feel like they move objects in the wrong direction. Try flipping the X-axis. [ ] Flip camera X-axis.

If your camera is behind you and over head, you may need to flip both the Y-axis and the X-axis.

[terracoda]

@BLFiedler I generalized the language a little bit.

[terracoda]

Actually, my version is way too long, let's try a very slightly tweaked version of @BLFiedler 's version

Troubleshooting Camera Input: Hands

The simulation assumes your camera is facing you as with a traditional laptop webcam. If your camera setup is different, use the options below to change the camera detection of your hands to match your hand movements.

Are your hands mapped to the wrong objects in the sim? [ ] Flip camera Y-axis.

Are your hands moving objects in the wrong direction in the sim? [ ] Flip camera X-axis.

Voicing context responses:

• Upon checking: Y-axis for camera flipped.
• Upon unchecking: Y-axis for camera no longer flipped.
• Upon checking: X-axis for camera flipped.
• Upon unchecking: X-axis for camera no longer flipped.

Note the other text will need to be read blocks for Voicing.

[terracoda]

@zepumph, I think this is ready.

[terracoda]

@zepumph, any questions, please ask. This was a monster issue. You only need to look at https://github.com/phetsims/ratio-and-proportion/issues/493#issuecomment-1320392707

[terracoda]

And, there is no artwork needed :-)

[emily-phet]

@terracoda @BLFiedler Here's a suggestion for less text, and leaning in to the specific reference to the "simulation hands". Bold is to emphasize where I made changes only, not intended for the end user. :)

Here's my thinking:

• I figured once we mention "hands" in the sim once, we can assume this is sim-specific text and we can keep on using it. So, I used "simulation hands" throughout.
• The camera recognition does not change based on these settings, only how what is recognized is mapped in the sim. So I tried to make the wording more aligned with that. Camera detection is not being changed. I also think it's less important to describe our default scenario and more important to indicate "here's where you make changes", so I shortened the text to jump straight to this sentiment.
• Under the Voicing context responses - I just suggested using "not" instead of "no longer" since it's shorter and feels more direct.
• I don't love the use of the word "troubleshooting", since it has a negative connotation, but I think I understand why this was chosen. I trust your judgement on it's use so my suggestion below has it in. :) A bit of an aside, but there is movement towards eventually having a direct link in the sim to the sim's Github repo, issue, and discussion features, so in the future we may be able to provide a more comprehensive troubleshooting guide for features in a location that is readily accessible from the preferences or current "PhET menu".

These are just suggestions, and @terracoda if you feel strongly about any or all of the original text, please go ahead with it. I just had some ideas that could shorten and perhaps simplify the text, and thought that might be helpful to share.

Troubleshooting Camera Input: Hands

If needed, use the options below to change the behavior of the simulation hands to match your hand movements.

Are your hands mapped to the wrong simulation hands? [ ] Flip camera Y-axis.

Are your hands moving the simulation hands in the wrong direction? [ ] Flip camera X-axis.

Voicing context responses:

Upon checking: Y-axis for camera flipped. Upon unchecking: Y-axis for camera not flipped. Upon checking: X-axis for camera flipped. Upon unchecking: X-axis for camera not flipped.