Review possible balance sound mapping examples for Screen 4

[Ashton-Morris]

I have created a video where I explain it. We can talk during next meeting but since we didn't get to it, I thought we could listen asynchronously here also.

https://github.com/phetsims/mean-share-and-balance/assets/34492870/cbdd41b4-b069-4818-9a6e-4133f85e3ce5

There are 6 options I made.

1. It gets more vibrato the further away from balance you are
2. It gets more vibrato and the pitch goes up the further away from balance you are
3. It gets more filtered (muffled) the more further away from balance you are (similar to screen 1)
4. It gets more filtered and vibrato the further away from balance you are
5. The tone plays less and less the further away from balance you are (which is reminiscent of bouncing or stretching)
6. The tone plays more and more the further away from balance you are (which is reminiscent of bouncing or stretching)

[amanda-phet]

Thank you for making this video Ashton! It is really easy to follow the examples with your included text.

My preference is option 3, with the filtered chord. I think it's really close as it is and I wouldn't want to change much. I also like the similarity to the first screen.

I think we could also go the direction of options 5-6, but might need some time to tweak it to get it right.

[marlitas]

I agree with @amanda-phet. 3 is my favorite, and since it connects to the first screen it might provide a cognitive bridge there.

I also really like 5 though. I do feel like I'm getting really good feedback for when I hit the balance exactly, while 3 feels like a spectrum a bit more.

I wonder if we could combine option 3 and 5 a bit? Maybe we have the high octave "ding" play when you hit the balance, but the general feedback of how close you are is set by the filtered chord option shown in 3?

[emily-phet]

I think for 1-3, it's hard to to know what is actually being sonified. We're seeing a physical setup, something that could be a board moving and sometimes hitting the ground (when way out of balance) but the sound has nothing to do with that.

What I was imagining before hearing the sounds was a kind of pleasant "creaking" sound as a the board is close to balance, and a pleasant (more success-like sound when in balance), and a distinct sound (not a physical thunk, but something reminiscent of that) when a side hits the ground. This maps to what you're actually seeing and aligns with people's physical experience of the world.

The closest to that for me of what Ashton tried was 5 or 6. I think we should avoid adding an abstract sound in cases where the design is intentionally using a physical, concrete, representation.

The filtered sound in screen 1 was designed to be reminiscent of water, because the physical representation we're using is a fluid. We would have gone with something even more water-like...except we've worked with water sounds and there are some technical hurdles to be able to get over some perceptual challenges with more water-like sounds, so we chose to avoid those in the interest of time and went with a more abstracted fluid-like sound.

[emily-phet]

OK - I spent about 30 minutes playing with the balance board/fulcrum, and considering sound mappings. What's key to start with is to considering where a person's attention is likely to be when they're interacting. So I started with a cognitive walkthrough (trying to figure out how to balance the board as if I know nothing about the sim) and throughout considering where my attention is. Then I laid out what seemed to me to be the most relevant categories of the balance board scenario (from a user's perspective), and suggested sound mappings for this. I also have a thought re input, noted below as well.

Here's my notes for doing all of this: Where is your attention during the task of balancing? EM’s thoughts: First, you need to figure out how to interact with the balance board situation. On screen start, there’s a cue to kick. You do that some, balls appear on the balance board, and arrows by the triangle. OK. You go click on the arrows (maybe they move something left/right), nothing happens. You click on the triangle, nothing happens.[1] (Only because I know…) I click and drag the triangle, slowly at first. Randomly I hear some click sounds, not sure what that means.[2] I drag faster, I hear more clicks, nothing happens. I click the “check” box. Hear an unpleasant “crash” sound, balance board left side hits the ground. [3] [if I didn’t know better, I would press reset and probably do that again, not knowing that I can move the triangle without the pillars present] but I do know better so I move the triangle and the balance board shifts from left side on ground to right side on ground. As I move more slowly I can get it to balance. As I do that, my eyes move back and forth along the balance board. I’m no longer looking at the triangle as I work towards balance, I’m looking back and forth along the purple balance line. This is where attention is, this is what the sound should be associated with, not the triangle movement but the angle of the purple line.

As that movement gets more smaller (closer to balance point) there should be “happier” sound, and at balance point a “success” or “completion” sound.

There are 2 states:

1) Board off ground - this one has two substates: 1a) Balanced 1b) Unbalanced (but still with board off ground), this is where there is likely to be more finer movements more of the time.

2) Board on ground (this is where you’re typically far from balance, and faster movement more likely)

We want there to be a distinct, happier, sound when you’re in the mode of “board off ground”. You’re closer to balance point.

Sound Design / Mapping Suggestion: there is a single tone played for discrete increments of board movement. When you’re moving fast you’ll hear these in quick succession, as you move slowly (close to the balance point typically) you’ll hear these less rapidly. These tones have pitch mapped to angle of the balance board (angles closer to parallel with the ground sound more “pleasant”, not sure if that’s pitch up or down…Ashton will probably have a better way of thinking about this. My guess is higher pitch = closer to balance).

When one side of the board touches the ground, there is a different set of tones played (maybe an octave lower in tone, so the sound is deeper = farther from balance?). In the “side of board on ground” state, you hear a tone corresponding with angle of the balance board, but the whole set of tones in this state are deeper than “board off ground” state.

Question re movement of triangle/input: when using a keyboard it takes ~20 presses to traverse the full span of the balance board. When you move really slow, you notice that you’re not really moving continuously with the mouse, but rather in very tiny discrete steps. If you move slowly with a mouse, it takes at least 5 times that. Could we increase the span of those discrete steps by double, making it so that –-when you move slowly—you encounter less “steps” before you find balance?

Interaction Issues noted in cognitive walkthrough: 1: [maybe the fulcrum should move some when arrows are clicked] 2: [how the click sounds are triggered is non-intuitive with cursor input, need to hear more on slower movement I think, and sound needs to trigger more consistently]

3. [instead of a "crash" sound, we should hear --if we go with my mapping suggestions-- the same tones play that you would, conceptually mapped to the angle of the board (not the movement of the fulcrum). (I know they may need to be technically mapped to different things, but conceptually, it's only the angle of the board that matters for the sounds I'm suggesting.]

[marlitas]

Question re movement of triangle/input: when using a keyboard it takes ~20 presses to traverse the full span of the balance board. When you move really slow, you notice that you’re not really moving continuously with the mouse, but rather in very tiny discrete steps. If you move slowly with a mouse, it takes at least 5 times that. Could we increase the span of those discrete steps by double, making it so that –-when you move slowly—you encounter less “steps” before you find balance?

This is easier to customize with keyboard because there can be a default step value and a "smaller intervals" step value (I'm not remembering the technical term we use right now). If we changed the step value for mouse we would have to round the mean according to that step value as well. Right now we're rounding to the nearest tenth. In the example you've suggested this would now make us round the mean to the nearest 0.2. For example 1.33333 === 1.4 instead of 1.3 or 1.5 === 1.6. I'm not sure that's ideal for the pedagogical goals of the sim.

[jbphet]

I'm also a fan of 3 and 5. 1 & 2 are a bit jarring with the vibrato. I like @emily-phet's suggestion that the sounds get "happier" sounding as we approach the balance point, and we can use the range where the beam isn't touching the ground for that.

[amanda-phet]

4/30 Discussion

We want to sonify the balance board, not necessarily proximity to the mean. We want to map the sound to the angle of the balance board.

We will use tone and pitch change to indicate proximity to success. We need a set of tones for when balance board is touching the ground, another set of ones when balance board is not touching the ground.

We might be feeling a disconnect between sound being used to enhance the visual experience and the sound being used to assist in achieving a pedagogical goal.

@jbphet and @marlitas will take a first stab at using the sound from option 5 from Ashton to map to the angle of the beam so we have something we can hear in context, then review it and discuss. We also want a success sound in.

Might consider query parameters if we want multiple options for interviews.

To try for iterative purposes:

• [x] Sonify the angle of the balance board
  • [x] Balanced should be the point of the highest pitch, and pitch will go down and the angle moves further from balance
  • [x] Shoot for a sort of creaking sound, composed of separate individual sounds played close together, and changing in pitch based on angle, but use existing sounds for now to get things started
  • [x] Try a short marimba sound as one of the sounds
• [x] For the case where the fulcrum is dragged and the pillars are in place, make the default sounds occur more often, specifically every 1/2 tick mark.
• [x] The fulcrum dragging sounds should be turned off when the pillars are gone so that they don't compete with the beam angle or proximity-to-mean sounds
• [x] For an alternative to sonifying the beam angle, have sounds that are driven by the fulcrum dragging, discrete sounds, and have the pitch go up as the mean is approached. Have the sound go up an octave when the mean is reached so there is a clear "success" indication. When the pillars are removed in this case, the sound for the location of the fulcrum is played.

[jbphet]

I've added code for the items listed above and a query parameter to support different permutations of sounds and algorithms. My apologies for the number of modes here, but there are a lot of potential variations, and I thought it would be good for us to experience a number of them. The query parameter is beamSoundMode and the possible values are 0 through 6. Here is a summary for each:

Setting	Description
beamSoundMode=0	Short marimba sound, triggered by fulcrum movement, pitch varies over entire range, peaks at mean.
beamSoundMode=1	Short marimba sound, triggered by fulcrum movement, pitch is the same until the beam starts to tilt and then gets higher, peaks at mean.
beamSoundMode=2	Creaky sound, triggered by fulcrum movement, pitch varies over entire range, peaks at mean.
beamSoundMode=3	Creaky sound, triggered by fulcrum movement, pitch is the same until the beam starts to tilt and then gets higher, peaks at mean.
beamSoundMode=4	Short marimba sound, triggered by changes to the beam angle, pitch is highest when beam is level
beamSoundMode=5	Short marimba sound, triggered by changes to the beam angle, pitch is lowest when beam is level
beamSoundMode=6	Creaky sound, triggered by changes to the beam angle, pitch is highest when beam is level
beamSoundMode=7	Creaky sound, triggered by changes to the beam angle, pitch is lowest when beam is level

Everybody please take this for a bit of a test drive prior to the meeting tomorrow, 5/2/2024, or review independently if you can't make it to the meeting. Also, please remember that we're reviewing the concepts here and not the sounds themselves - I've used preexisting sounds for everything, and we expect to fill those in with more appropriate ones once we've decided on which concept(s) we prefer.

[marlitas]

Some initial thoughts:

• My favorites so far is 1 although I also like 7 with some caveats and potential adjustments.

• What I like about 1 is that it doesn't lead you to the answer straight away, and hits that combination of beam tilt and proximity to the mean that I believe @emily-phet was talking about. It also gives you a notion of small success when the note starts to change, which I think hits part of @amanda-phet's goal that students should feel success even if they're a tenth or two off. They pretty much nailed it, and the sound is helping confirm that for me with soundMode 1.

• I like that 7 gives you this sense of climax right before you hit the mean on either side. I think it also helps with @amanda-phet 's goal that I mentioned above, while also illustrating the physicality of the beam. A couple of concerns:

  • I had a hard time hearing the difference in pitches when the mean was closer to center. I think it would help if they could be a bit more drastic in those scenarios? Compared to when the mean is on the edge that "climax" point is very clear.
  • I experienced this sense that if I could move the fulcrum in the opposite direction to infinity I would also find balance... and that was interesting, but also didn't feel like it was relevant or part of the pedagogical goals of the sim. We could perhaps address this by having the sound start to fade away the further you get from the mean as well? This would maybe help address both behaviors we're wanting the sound to capture.

I'm sure we'll talk more in the meeting, but just some initial thoughts to jot down :-)

[amanda-phet]

Looking forward to discussing. Some thoughts about the options:

Options 1 and 3 have a lot of time spent on the same sound, but the board is actually moving a bit, so it feels disconnected to me.

Options 4-7, based on beam angle, have an unfortunate effect that when the board is no longer touching the ground, there is an inflection point in the sound, which brings attention to something that shouldn't really be grabbing your attention in that way. Yes, you're closer to the mean, but there isn't something actually special about that point, we just picked a location where we wanted to visually indicate that you're close to balance to reinforce that your prediction was close (because being precise to the nearest tenth isn't the point).

Option 0 is probably my favorite, even though I didn't think I'd like something that simple based on proximity to the mean.

[Ashton-Morris]

Thanks for putting these together before the meeting! Overall I thought that option 0 represented the proper balance (get it:) between teaching the learning goal and being aesthetically pleasing also.

I was really hoping to hear a version with the type of stretching mapping I had showed in my fifth and sixth example, but I understand that we are short on time.

[amanda-phet]

Spent a good amount of time discussing this. We are going to go ahead with option 7 for interviews. We like that it gives feedback immediately upon removing the pillars, as well as when moving a ball. We'd like to see how students interpret the fluctuation of the pitch that happens when close to the mean.

[amanda-phet]

Discussed 5/14

• Ashton says he could make 5 creak sounds and we'd probably still end up at the one we are currently using
• Emily says it might be too many clicks at the current rate.

We tuned the rate a bit and are happy with it.

Over to @jbphet to commit the changes and then I can do a final review!

[jbphet]

When closely reviewing the behavior for the option that the sound design team has chosen, I noticed a problem: The sound generator isn't always playing the lowest creaking sound when the plank is level. I did a little investigation and I know why. The code is set up to play a sound when the angle changes, then hold off on allowing subsequent angle changes to trigger a sound generation (a single creak sound) for a short period of time. For small angle changes near the mean, this has the affect of playing a sound at the start angle but not the finish angle. So, if you start near the mean and move to it, you hear a sound for the angle near the mean but not at it. If you start at the mean and move away from it, you hear and sound for very close to the mean but not at the final angle.

I'd like to think on this for a bit before deciding how to potentially fix it.

[jbphet]

One idea would be to play the sound at fix intervals rather than based on a timeout.

[jbphet]

I discussed this with @amanda-phet, and she suggested we try not playing the creak sound at all at the median, and letting the 'success' sound play by itself. Seems quite reasonable to me. I've implemented that behavior, and it seems quite good. I also slightly reduced the pitch variation for the tilt of the balance beam because I felt like it was a bit too much.

@amanda-phet - I think this is now ready for you to review.

[amanda-phet]

This is working great! I like the change to how the sounds are working when you successfully balance the beam. I liked the pitch before, but the new one is fine, so I don't think we need to keep discussing that.