search

phetsims / wave-interference

"Wave Interference" is an educational simulation in HTML5, by PhET Interactive Simulations.
MIT License
18 stars 5 forks source link

Sound: Implement new slider click sounds and get feedback #484

Closed Ashton-Morris closed 4 years ago

Ashton-Morris commented 4 years ago

I spoke with Ariel and we liked the subtlety of the step forward button.

So here is a shortened version of that to try

click-right click-left

Ashton-Morris commented 4 years ago

@samreid I can't add assignees so hopefully you are notified of this.

samreid commented 4 years ago

Since this will be implemented in the "wave interference" repo, I'll move the issue there for tracking.

samreid commented 4 years ago

I added the proposed sounds. @ariel-phet or @Ashton-Morris can you please review?

ariel-phet commented 4 years ago

I think the sound needs to be volume balanced, but it certainly seems much less "geiger counter" to me. I think the sim as a whole needs to be reviewed once volume balanced, but to me this issue solved the main problem I was hearing.

emily-phet commented 4 years ago

  1. (most important!) The sounds I hear when I move the slider are more “click"-y (and less "bump"-y) than I remember form the last design meeting that @jbphet or @Ashton-Morris demoed. @samreid and @Ashton-Morris - can you confirm that this is latest sound for the sliders - Here's the sound I hear (note - slight warping at the end of each click sound is an artifact of my recording this as it is played from my computer speakers). Let's make sure we're all considering the correct bump-y sound (and not the default click-y sound in common code) before proceeding.

  2. Timing - Also I wish the timing of the clicks sound felt more consistent with the visual tick marks across the slider range. It lines up nicely when moving slowly, but feels slightly off at times when moving more quickly. Is there any way to associate the click sound with the literal visual location of the slider/grid marks? I feel like we’re discussed this before, but I think the more associated the click sound is to exactly the grid marks on the slider, the more “intuitive” the sound will feel and less attention-grabbing. @jbphet - thoughts on this?

  3. End sounds - one thing that might help us out in tweaking the sounds so that they feel more like they are associated with the tick marks is to have a slightly different sound for the extreme ends of the slider (when you've "hit the edge" so to speak). @Ashton-Morris - once we are clear on which version of the sound is the right one to move forward with (the bump-y one!), please make a variant that is a subtle indicator max/min has been reached. I imagine max/min can be the same sound, but if you think it's better to have these be slightly different, please go ahead.

  4. Volume - I think if the volume is decreased further in the final sound balancing (these should be barely noticeable at medium volume), it would help. Let's not deal with that in this issue, though. It will depend if there is a mix-up on the sound timbre, a "bump"-y sound may need to be slightly louder than a more "click"-y sound to be perceived as the same volume.

samreid commented 4 years ago

Another issue about the slider "clicking" sounds: on the water drop screen, the slider clicks sound kind of like water drops: may be a little confusing.

jbphet commented 4 years ago

The sounds I hear when I move the slider are more “click"-y (and less "bump"-y) than I remember form the last design meeting

I just listened to the current master version, and I'm pretty sure it is the same sound. It sounds less "clicky" in headphones to me, more so over my computer speakers.

[T]he timing of the clicks...feels slightly off at times when moving more quickly...@jbphet - thoughts on this?

I just tested this, again using the current master version, on Chrome and Firefox on my Win 10 machine, and I don't find the timing to be noticeably off when moving fast. This might be a browser-specific thing. @emily-phet - can you let us know what browser and OS you're using when you see this, and can you test whether it occurs on other browsers on that same machine.

My thoughts are that I would have to duplicate this, make some measurements, and see what changes could be made to improve it. Different browsers may be handling the situation where there are a lot of sounds to play at nearly the same time differently, and we may be able to make some adjustments to improve the performance in such situations.

emily-phet commented 4 years ago

(Updating my previous comment after discussion with @jbphet)

  1. (most important!) The sounds I hear when I move the slider are more “click"-y (and less "bump"-y) than I remember form the last design meeting that @jbphet or @Ashton-Morris demoed. @samreid and @Ashton-Morris - can you confirm that this is latest sound for the sliders - Here's the sound I hear (note - slight warping at the end of each click sound is an artifact of my recording this as it is played from my computer speakers). Let's make sure we're all considering the correct bump-y sound (and not the default click-y sound in common code) before proceeding.

With @jbphet it's clear I'm hearing the correct sound. We concluded that my impression of it during last week's meeting was influenced by hearing the original click sound...comparing the new sound to the old, the newer one definitely sounds 'bump'-y, but I think we can still improve upon it. @Ashton-Morris can you please propose one or a few variations of a 'bump'-y sound, rather than a 'click' sound. We can discuss further in tomorrow's meeting if that would be helpful. Whatever we go with, we'll also want sounds specifically for the max/min points (from 3) below).

  1. Timing - Also I wish the timing of the clicks sound felt more consistent with the visual tick marks across the slider range. It lines up nicely when moving slowly, but feels slightly off at times when moving more quickly. Is there any way to associate the click sound with the literal visual location of the slider/grid marks? I feel like we’re discussed this before, but I think the more associated the click sound is to exactly the grid marks on the slider, the more “intuitive” the sound will feel and less attention-grabbing. @jbphet - thoughts on this?

@jbphet had a number of suggestions as to why the timing might feel off, and why the sound may be perceived as amplified or more noticeable when the slider is moving quickly (I believe he'll list these in a later comment).

  1. End sounds - one thing that might help us out in tweaking the sounds so that they feel more like they are associated with the tick marks is to have a slightly different sound for the extreme ends of the slider (when you've "hit the edge" so to speak). @Ashton-Morris - once we are clear on which version of the sound is the right one to move forward with (the bump-y one!), please make a variant that is a subtle indicator max/min has been reached. I imagine max/min can be the same sound, but if you think it's better to have these be slightly different, please go ahead.

See 1) above.

  1. Volume - I think if the volume is decreased further in the final sound balancing (these should be barely noticeable at medium volume), it would help. Let's not deal with that in this issue, though. It will depend if there is a mix-up on the sound timbre, a "bump"-y sound may need to be slightly louder than a more "click"-y sound to be perceived as the same volume.

No update on 4) to add.

Unassigning @samreid for now, as I think there's some things here for me, @Ashton-Morris and @jbphet to sort out for the moment.

jbphet commented 4 years ago

In the previous comment, @emily-phet mentioned that she and I discussed why the slider sounds a bit odd and artificial. I think a possible cause of this is that the sounds are initiated only at very specific times based on the 60 frames/sec rate at which the browser runs. As a result of this, sound can end up with the same phase relationship all the time, which can add together and boost the volume. In a situation where a lot of sounds are being initiated at once, such as when the slider is moving very quickly, it may even be that the sound clip is being initiated more than once in a given animation frame. There are several things that I can think of that might serve to address this.

Once @Ashton-Morris has created the new sounds and they have been integrated, @samreid and I can collaborate and try some of these out and find which ones are the most effective at addressing the issue.

Ashton-Morris commented 4 years ago

I have experimented with some more slider clicks for everyone to listen to. I have individual files but I also recorded a few examples of how it would sound dragging it from left to right. Hear below.

slider-clicks-idea-a-example.mp3 slider-clicks-idea-b-example.mp3 slider-clicks-idea-c-example slider-clicks-idea-d-example.mp3 slider-clicks-idea-e-example.mp3 slider-clicks-idea-f-example.mp3

jbphet commented 4 years ago

I listened to each of the sounds that are linked in the previous comment and below is how I would rank them, with notes on each. I listened on headphones and did not evaluate on any speakers. These are my opinions, your mileage may vary.

  1. idea-c - This is similar to the current sound but nicer, a bit more of a "bump". There are two end sounds in the example, and I think we should just use one to signify an endpoint, and my preference would be the 2nd one.
  2. idea-d - Also pretty similar to the current sound, but better. I just like c a little more.
  3. idea-f - This one is a little much, but might work if the volume is set pretty low.
  4. idea-e - A little too "toneful". Sounds kind of like a slot machine.
  5. idea-b - A bit too game-like and would thus bring more attention to the sound than I think we're after.
  6. idea-a - Sounds too much like a small laser being fired in a video game.
jbphet commented 4 years ago

Adding @samreid to the list of assignees. If everyone can comment on which of the latest sounds they like best, I'll add the top three or four as options for this week's review process.

samreid commented 4 years ago

For me, a,b,f,d seem too percussive. "c" seems like a balance between not too percussive to be annoying, but still succinct enough that it cues dragging across each tick. I agree with @jbphet that "e" is too toneful and game-ish.

In a radio button group, there is a pitch shift between each successive option. That seems like it would be a good audio cue here as well for what is happening. Not just telling you "you took a step up or down" but telling you "you took a step up or down and here's where you are now." Can we try mocking up a pitch shift on each successive tick mark? Or will a quick drag be too cacophonous?

samreid commented 4 years ago

Also, have we tried a less "full" sound, like playing cards in bicycle spokes? Maybe like the and last couple of seconds of https://www.shutterstock.com/video/clip-2065526-playing-card-spokes-bicycle (in the middle 3 seconds to 25 seconds, it is too rapid to be useful.)

Ashton-Morris commented 4 years ago

I feel like playing cards would end up sounding too much like a geiger counter like the other clicks.

I'm going for more of a bump as @emily-phet suggested but a lot of these issues might be solved with the implementation and amount the sound is allowed to play when sliding.

emily-phet commented 4 years ago

Here are my thoughts:

C - favorite, particularly for sliders with visible tick marks. May want to consider different variant of this for sliders without tick marks, or with few tick marks. E - kind of like C, but has a bit of an undesirable “space-y’ or ’sci-fi’ feel D - still a little too shooting game-y to me, but perhaps could be softened a bit or something to emphasis more of a ‘bump’ feel than a ‘shooting’ feel. F - too physical sounding, but could work for a different purpose in a different sim. I like how it sounds like a cartoon-ized mechanical sound. A - too shooting game-y B - too chirpy

Seems like there's a lot of interest in C. If there's any way we can get a working version of that in a slider before today's afternoon meeting (in Waves or some other one if that's easier) would be awesome!

emily-phet commented 4 years ago

Regarding the pitch change to cue increase/decrease of slider that @samreid mentioned, I think @samreid is spot on in general - but not for this slider sound. The specific slider sound we're designing in this issue is explicitly intended to not be one of the top sounds the learner is being scaffolded to attend to. It's intended to serve as secondary feedback, and needs to be relatively neutral. If folks want to discuss this further, let me know, and I can describe specific examples that have led us to this general conclusion, based on 1) various studies that we've completed on how many and what types of sound mappings work well together, as well as 2) wanting to create standardized UI sounds (such as this slider sound) that can be used across many sims without re-design.

samreid commented 4 years ago

I committed the "C" series sounds for right-dragging and and reduced it by one semitone for left-dragging. It can be reviewed on phettest for today's meeting.

jbphet commented 4 years ago

I added a boundary sound based on @Ashton-Morris's idea-c example clip, which can also be reviewed at the meeting.

Ashton-Morris commented 4 years ago

Here is a minor third version of idea c

@jbphet

jbphet commented 4 years ago

We reviewed the latest in today's meetings, and we agreed that "idea-c" sounds good in context. Next steps:

I'll implement these before the next meeting.

jbphet commented 4 years ago

I've implemented all three items from the list above.

The third item involved testing whether a sound was being played in the same animation frame. I added some debug code to test whether this was happening, and found that while the playing of the clicks was often less than a frame time (1/60=0.016667 seconds) apart, they weren't occurring on frame time boundaries. I think this means that Scenery input events are no longer batched, and I have a question out about this. I was seeing cases where the play requests were as little as 2 ms apart, and on fast slider moves differences of 4 to 8 ms were fairly common. I went ahead and added code to prevent playing sounds closer than 1/60 seconds apart, but if the events aren't batched, this value is rather arbitrary and can be anything we want based on what sounds best (if we end up wanting it at all).

I personally don't notice a big difference when this is in place, but I'm happy to leave it in if others find it to be an improvement.

emily-phet commented 4 years ago

@jbphet How do I hear the latest? I listened to what I thought was the latest on phettest, and it sounded like the squishy button and slider sound from last week, not the new ones...

jbphet commented 4 years ago

@jbphet How do I hear the latest? I listened to what I thought was the latest on phettest, and it sounded like the squishy button and slider sound from last week, not the new ones...

I just checked, and the updated sounds are playing on the master version for me. The button sound is one of the sounds that we had in place last week, but it wasn't the default, and the differences for the click sound are somewhat subtle, since I made it so that the move-right and move-left sounds are the same, added new end sounds, and prevented overlap playing. Or maybe phettest isn't updating correctly.

@emily-phet - if you're available during today's sound design meeting, let's have a listen then. If not, we can set up a time to meet on Zoom.

Ashton-Morris commented 4 years ago

@emily-phet I wasn't always hearing new sounds in phettest either. It might be a caching issue that I had too.

In Chrome there are three dots on the upper right. Click that then - more tools then - developer tools. There is a way do disable caching while the developer tools are open. I can't remember though @jbphet might know.

But doing that has helped me hear the latest versions.

jbphet commented 4 years ago

We reviewed the latest implementation in today's sound design review meeting, and @emily-phet and @Ashton-Morris agree that this sound behavior is good.

Assigning to @samreid to follow up on handling for keyboard-based interaction. The current code looks like it will only handle tick mark crossings, and it needs to be able to generate a sound on each value change when triggered by a keyboard action (this may be there and I was just missing it, I'm not certain).

samreid commented 4 years ago

@jessegreenberg or @zepumph we would like to play a sound effect when the slider value is changed by a keyboard press. There is already a different sound effect when the slider thumb is dragged across a tick mark. How can we listen for the keyboard press?

jessegreenberg commented 4 years ago

I pushed a proposed a change for this in https://github.com/phetsims/sun/issues/589. @samreid if you pull sun, you should be able to have sounds from alternative input by adding logic under an event.isFromPDOM() check in your drag function. It would look something like

      drag: event => {
        const value = property.value;

        if ( event.isFromPDOM() ) {
          // generate a sound once per event from alternative input
          ...
        }
        else {

          // handle the sound as desired for mouse/touch style input
          for ( let i = 0; i < ticks.length; i++ ) {
            const tick = ticks[ i ];
            if ( lastValue !== value && ( value === property.range.min || value === property.range.max ) ) {
              sliderBoundaryClickSoundClip.play();
              break;
            }
            else if ( lastValue < tick.value && value >= tick.value ) {
              sliderIncreaseClickSoundClip.play();
              break;
            }
            else if ( lastValue > tick.value && value <= tick.value ) {
              sliderDecreaseClickSoundClip.play();
              break;
            }
          }
        }

        lastValue = value;
      }

Happy to talk more if its helpful!

samreid commented 4 years ago

Thanks @jessegreenberg. In order to play a different sound when at the extrema, I needed to wait until I have the final property value, and also to know the direction of the change. Can you please review the change and help me make it more robust? What if other keyboard events cause the slider thumb to move?

samreid commented 4 years ago

I made more commits that eliminate the keycode problem. But the two remaining problems are:

jessegreenberg commented 4 years ago

Are we guaranteed that drag will be called before the property value changes?

This seems like a very deliberate decision in Slider.js, I assume this not going to change.

I wonder if we need to make sure that using the keyboard puts it at 1.00000 instead of 0.9999999 in the first place, so there's no discrepancy there. How could this be done?

Could we use constrainValue for this, something like

constrainValue: value => Utils.toFixedNumber( value, 5 )

I don't see a better way currently within the drag callback to get what the value is going to be and get around the timeout. We could potentially add a new callback option to Slider.js and AccessibleValueHandler.js to support this. But maybe it would be acceptable to have sound from alternative input happen on endDrag?

      endDrag: event => {
        if ( event.isFromPDOM() ) {
          if ( property.value === property.range.min || property.value === property.range.max ) {
            sliderBoundaryClickSoundClip.play();
          }
          else {
            sliderClickSoundClip.play();
          }
        }
      }

The downside of this is that you wouldn't get any sound while pressing and holding an arrow key, only on release.

zepumph commented 4 years ago

Are we guaranteed that drag will be called before the property value changes?

This seems like a very deliberate decision in Slider.js, I assume this not going to change.

It looks like it was added as part of https://github.com/phetsims/sun/issues/561

At first glance it seems quite buggy to have the callback before the Property changes. I commented over in https://github.com/phetsims/sun/issues/561#issuecomment-614929991 about that.

Everything else here is looking very nice!

samreid commented 4 years ago

Blocked until https://github.com/phetsims/sun/issues/561 is improved.

emily-phet commented 4 years ago

@jbphet I'd like to discuss today any tweaking we can do to decrease the total number of slider sounds heard when moving the slider very fast.

jbphet commented 4 years ago

We made a mod in today's sound design meeting, @emily-phet will try it out and report back.

emily-phet commented 4 years ago

@jbphet - Checked latest in phettest - changes from today sound good to me! Let's go with that. Next step would be a final sound volume mix check-off from @Ashton-Morris, and then send out to full team for final input (ideally on sound volumes only, limiting other feedback to only if absolute deal breaker for moving forward for publication).

Ashton-Morris commented 4 years ago

I reviewed the levels today and determined that the were actually spot on! No changes were needed from my perspective.

@emily-phet You can get this to the full team for final input.

emily-phet commented 4 years ago

Off to final feedback from the team. Closing this issue, as feedback will be requested on the sim as whole, not just the slider sounds.

jessegreenberg commented 4 years ago

Reopening for the remaining questions about alternative input. https://github.com/phetsims/sun/issues/561 is done so this is no longer on hold and we should be able to remove the timeout.

The other question from https://github.com/phetsims/wave-interference/issues/484#issuecomment-614266199 was

I wonder if we need to make sure that using the keyboard puts it at 1.00000 instead of 0.9999999 in the first place, so there's no discrepancy there. How could this be done?

Can constrainValue be used for this?

samreid commented 4 years ago

I used constrainValue and the new slider interface, works great! Thanks @jessegreenberg, closing.