Initial sound design for Quadrilateral

[jessegreenberg]

We are beginning to work on sound design in this sim. In discussion today we decided to add two Properties to the model that represent the difference in oposite angles for the quadrilateral, being angle1DiffAngle3Property and angle2DiffAngle4Property. When both of these are below a threshold angle, the quad is considered to be a parallelogram.

We could begin to tie sounds to these two Properties, @Ashton-Morris can you please let me know when they are ready or if there is any other logic that will be needed to support the sound design?

[Ashton-Morris]

Initial Audio Mockups: Currently we are focused on parallels, length, and angles. Here is how my mockup is currently working.

• If sides 1 and 3 are parallel they play the same note. If they get longer or shorter in length but are still parallel they fade into lower and then higher octaves of the same note.

• If sides 2 and 4 are parallel they play the same note. If they get longer or shorter in length but are still parallel they fade into lower and then higher octaves of the same note.

• If sides 2 and 4 stay parallel but the angles C and D change, they both play the same note but higher in pitch. The more angle C is obtuse and D is acute the higher in pitch it goes, up to an octave or 12 semitones.

• If sides 2 and 4 stay parallel but angles C is acute and angle D is obtuse they play the same note but lower in pitch. The more they lean to the right, the lower in pitch it gets up to an octave below or -12 semitones.

• If sides 2 and 4 do not stay parallel then they do not play the same note. The note or tone they play is based off of the angles C and D.

• Angles C and D work opposite in how they map to tones. This allows them to play the same tone in parallel.

There are two different ways to do the audio but they both essentially map the same way. There are just more steps to the Chromatic version.

• Chromatic https://streamable.com/th79qc
• Non Chromatic https://streamable.com/y6tyg7

The Chromatic version has the files fading into and out of each other as they go up and down in scale. Visually that looks like this in my software. (the white angled lines are the fade points)

[jessegreenberg]

Looks good @Ashton-Morris, thanks for writing this up. This definitely helps for my understanding of it. I could start adding information to the model to support this (Properties for pairs of parallel sides and Properties for side lengths) once we get a go-ahead from @emily-phet @BLFiedler or others.

[emily-phet]

@jessegreenberg - I think we need to consider Ashton's description of the sound mapping above, and translate that into terms that use 1) side length (to change across octaves) and 2) the variable we discussed yesterday comparing opposite angles. I think this is possible, and would stay more closely aligned with the definition of what a parallelogram is.

Regarding which of the two sound design variants to implement - @jessegreenberg can you get started and provide an update when you have a sense of how much more challenging the chromatic vs non-chromatic approach will be? The chromatic one sounds more dynamic which may result in something more pleasing to the ear, but if it's a lot harder to get working the non-chromatic one would give us a chance to try out the mappings and from there we could decide if the more complex variant is worth the effort.

[brettfiedler]

Quick paste in of Jenna's comments in Slack so they don't disappear into the ether:

Ashton Morris I listened to both of the mock-ups. Two quick realizations I had: 1) I need more of this in my life! (there was something very calming about watching/listening to these) and 2) I have a very untrained ear. Then, I read the descriptions that you typed out. Wowza! I'm in the same camp as Brett - I found Mockup 1 a bit more pleasant to listen to overall, truly I like watching/listening to it - the sound of reverberation or something of that nature was a nice effect - didn't sound quite as sharp as Mockup 2. Again - purely qualitative impression from an untrained ear that taught piano for years but no longer gets to play much, so there's that. I tried to listen to these with and without my eyes closed, but I couldn't make the mapping in my head of what was happening. After reading the descriptions you typed out, I think it might have been for two reasons: 1) I expected (though perhaps not correctly), there would be a more differential/steeper gradient as the sides were stretched. It somehow felt like I was visually seeing more stretching/changing than I was hearing (particularly in the initial stretches where both vertices moved. I found it better with the tonal up/downs when the 2 vertical sides were no longer parallel). 2) There's a lot going on as we anticipated in our meeting discussion last week - the whole "whether or not 2 sides were parallel or not" wasn't coming through to me via the audio, but I also wasn't "listening" for that if that makes sense. It's like I tricked myself into anticipating what I thought I should be hearing in some weird way. I don't much appreciate feedback that doesn't give tangible steps forward :slightly_smiling_face:, so I'm trying to decipher my own thoughts into "what could we try next". I'm curious if others felt there needed to be a steeper gradient/change in audio with the changes that were visually observed when the lines remained in parallel? If yes, we could play with the scaling. If not, it's my untrained ear to blame. Might we want to think about separating the audio of the movement of the sides with whether or not the sides are parallel? Almost like, once the side length movements are complete, then we make some type of "goal state" sound that indicates parallel-ism - ha!? My two cents to keep the conversation going. Very much enjoyed toying with this, and it gives me even more inspiration to develop easily "feelable" haptic demos!

[brettfiedler]

Listened a few more times now. Upon considering Ashton's mapping above, I actually find that I prefer the Non-chromatic. As I mentioned, the chromatic is more pleasant, but I am still having trouble catching the mappings while things are moving around with the chromatic. It's a lot more distinct to me in the non-chromatic (and I think this is will remain true for me even if we play around with the range of pitches).

I'll echo Emily's suggestion of getting up and running whatever is low-cost time-wise/effort-wise, but I'd also possibly like to consider that we try a version that I think is the simplest possible soundscape for our task: one that isolates the "parallel" nature. I think for this, that means having a version that removes the angle and length modulations?

[jessegreenberg]

I read the sound documentation in tambo. I also reviewed BalloonVelocitySoundGenerator, which was really helpful and well documented. I have a better sense of how to implement this now.

However, I don't fully understand why the chromatic version will be harder to implement. @Ashton-Morris by "more steps" in the chromatic example do you mean more steps in the mapping for playback rate to change the pitch? If so, I would expect them to be the same amount of effort to implement. Or did you mean that there are more technical steps required to make the sound?

Also, I am wondering if the labels are correct in https://github.com/phetsims/quadrilateral/issues/4#issuecomment-915555926? Maybe I don't understand the sounds yet, but the "Non-chromatic" example actually sounds like it is going up a chromatic scale while the "chromatic" example sounds more like whole tones to me.

[jessegreenberg]

Actually, maybe the image in https://github.com/phetsims/quadrilateral/issues/4#issuecomment-915555926 is showing that this would be implemented with multiple sound clips that fade in and out of each other and so it wouldn't be mapping to pitch but to volume and the right sound clip to play?

[Ashton-Morris]

@jessegreenberg The non chromatic is labeled that because it slips in-between notes and on notes as it goes up and down (like a slide whistle). But the chromatic one skips over tones that are not in a chromatic scale (more like a flute) effectively making it seem like any two positions will sound somewhat harmonic and probably more pleasant.

The non-chormatic version would be easier because you would map an angle parameter to the playback speed/pitch of a sound source. In the chromatic version you'd first create the ability to fade into and out of multiple sound sources (or the same sound source played at different playback rates) and then map that to the angle parameter.

My guess is that the code used to make the lengthening sides turn into lower octaves can be used to create the Chromatic angle parts as well so maybe it wouldn't be harder after all.

[Ashton-Morris]

Actually, maybe the image in #4 (comment) is showing that this would be implemented with multiple sound clips that fade in and out of each other and so it wouldn't be mapping to pitch but to volume and the right sound clip to play?

Yes that one would be mapping to volume and the pitch would be pre chosen by the clips we select.

[jessegreenberg]

OK thanks!

[jessegreenberg]

In the above commit I added model components for sides, so they are aware of their lengths to support sound design. They are also aware of their slopes, so if "determine if two pairs of sides are parallel" is ever needed we can easily use that.

[jessegreenberg]

From https://docs.google.com/document/d/1OMU2DGyN8F_QGo_CkiAy4qCBeFe8BOOvA9ANzH7ZEbM/edit#

Ashton will provide sound clips for Jesse to add. We will see how they sound together and then work to fade them so they sound good/as designed

@Ashton-Morris reassigning to you for this, please toss it back to me when ready!

[emily-phet]

Just a note to @jessegreenberg - Let's discuss the model components in next week's Virtual Model meeting. I have some questions about the use of bottom/top and left/right, and slope that I'd like to talk through.

[jessegreenberg]

Sounds good! I believe they will be used to support the proposed sound design, which I might be able to add before next Tuesday if the assets are ready. If you would like to discuss before then let me know.

[Ashton-Morris]

@jessegreenberg I have added four files to this Audio folder. for the main sound. We can compare which sounds best.

• quad-loop-04.mp3
• quad-loop-01.mp3
• quad-loop-02.mp3
• quad-loop-03.mp3

John has some looping documentation somewhere. We used it for Gravity Force Labs Basics for Force if you need to look through the code.

Also since the same files will be playing simultaneously it would be best if they started at different start points so they would sound less repetitive. I would imagine if possible it would sound something like this in code: Play file as loop - start playhead or playing file at random point of the files duration.

This way if you had 4 same the file playing as a loop at the same time, they would all start at different times within that loop and the audio would have much more variation to the ears.

[jessegreenberg]

We reviewed the stat of design in a 9/21/21 meeting. Things that need to be done with the current prototype at this point:

• Fade in and out of the sound when it starts/ends so that it isn't as harsh.
• Fade through sound clips as we go up/down the scale when side tilt changes.
• For sound sources other than sound 1 (moving sound sources) start sound at a random point in the clip(?), see https://github.com/phetsims/quadrilateral/issues/4#issuecomment-921907219

@Ashton-Morris will also consider a different design and possibly create a mockup for it for next time. One suggestion was to map sounds only to pairs of parallel sides so that there are two sounds instead of four. Another suggestion was to only have a "success" sound that plays when a pair of sides become parallel.

[Ashton-Morris]

During the meeting today I showed some AV Mockups for the Sonification with a focus on the Parallel nature of the sides (only two sounds as opposed to 4 - one for each side). I have exported them for others to view.

• quad-parallels-mockup-no-volume-if-out-of-parallel.m4v
• quad-parallels-mockup-no-volume-if-out-of-parallel-with-pitch-changing-angles.m4v
• quad-parallels-mockup-staccato-state-changes-with-pitch-var-based-on-out-of-parallel.m4v
• quad-parallels-mockup-with-vibrato-based-on-out-of-parallel.m4v
• quad-parallels-mockup-wobble-with-no-sound-until-out-of-parallel.m4v

[jessegreenberg]

I built a new version with improvements to the sound design discussed in https://github.com/phetsims/quadrilateral/issues/4#issuecomment-924337371:

https://phet-dev.colorado.edu/html/quadrilateral/1.0.0-dev.4/phet/quadrilateral_en_phet.html

[jessegreenberg]

We discussed https://github.com/phetsims/quadrilateral/issues/4#issuecomment-929654068:

The favorites are

• "quad-parallels-mockup-no-volume-if-out-of-parallel.m4v" : This is the simplest option, and would be the quickest to add.
• "quad-parallels-mockup-staccato-state-changes-with-pitch-var-based-on-out-of-parallel.m4v" : This one is a bit more complex but was a favorite for those in the meeting today.

@jessegreenberg will implement these before next week though we way want to try others in the future.

Everyone will test the prototype in https://github.com/phetsims/quadrilateral/issues/4#issuecomment-941122356 and after playing with it we will discuss improvements for that design.

[Ashton-Morris]

I built a new version with improvements to the sound design discussed in #4 (comment):

https://phet-dev.colorado.edu/html/quadrilateral/1.0.0-dev.4/phet/quadrilateral_en_phet.html

As far as aesthetics go, I liked Sound One and Sound Three the best. Also the subtle touch of fading in and our the audio when you touched and released it sounded better.

Overall though to me the mapping didn't feel like it "meant" anything. I tested it with four fingers and sometimes I would make pleasant sounds and sometimes unpleasant sounds but they didn't relate to any sides being in parallel or not. Because of the mapping they just had the possibility of being pleasant or unpleasant. I'll brainstorm how to improve this.

[brettfiedler]

Roping @terracoda and @jbphet into this issue.

I mocked this up for everyone to reference while testing the sim as a whole, but was waiting for the quick change to multitouch to rope the whole quad channel in. However, it might be helpful for anyone testing the sound right now to get a sense of what things we'll be asking people to do, so you can better make sense of what the sonifications are trying to achieve:

Please consider the following tasks to guide your interaction (try to accomplish as quickly and smoothly as possible where you can):

• Create parallelogram shape(s). Move different vertices and sides to make different shapes and return to a parallelogram.
• While maintaining a parallelogram shape, transform the current parallelogram into any other parallelogram
• While maintaining a parallelogram shape, transform the current parallelogram into a mirrored parallelogram
• While maintaining all side lengths, transform the current parallelogram into a mirrored parallelogram
• If possible with your input method, using two hands (or two contact points), and while maintaining all side lengths, transform the current parallelogram into a mirrored parallelogram

[brettfiedler]

(re: 1.0.0-dev.4) This is a lovely music tool and I think we should publish this elsewhere for people to have that experience (1 & 3 sound good aesthetically to me as well. 3 definitely for the thriller ambiance).

That being said, I do think this particular sonification scheme (mapping pitch to side lengths) is pushing me not toward creating parallelograms in general (maybe only as an intermediate step), but very specifically to a square where the tones all seem to "line up" and in some cases resolve to some "pure" sound.

That there can be either really high tension or some discordant intervals while I have successfully made a parallelogram shape feels counter to the task as well.

edit: Another thought: I find myself avoiding small shapes and making the lines longer because the lower pitch is more pleasant to listen to even when there are discordant tones.

edit edit: Oh, duh. Sorry. The sound mapping is also on the angle? Not just the side lengths. I realized I was not able to successfully do the "Maintain all side lengths and transform into a mirrored parallelogram" in this scheme since the pitch on each angle necessarily changes as they pass through 90 degrees and invert. So - that is another thing to keep in mind about this scheme.

[jessegreenberg]

I made a new version with the sound designs in https://github.com/phetsims/quadrilateral/issues/4#issuecomment-941330768 added to the sim.

https://phet-dev.colorado.edu/html/quadrilateral/1.0.0-dev.5/phet/quadrilateral_en_phet.html?showVertices

[Ashton-Morris]

I spent some time trying other ideas this past week and wanted to post the videos for our record.

• quad-success-moving-in-parallel-lock-unlock-example.m4v This one has a kind of etherial success loop playing while moving in parallel. Then an unlock and lock sound when changing the shape out of parallel and a lock sound when going back into parallel.

• quad-rusty-gears-out-of-parallel-example.m4v This is an extreme version but it gets the point across. When moving while in parallel the gears have a clicking + a dinging sound to them. When moving out of parallel the gears start to sputter more and more like a broken machine but return to normal the closer you are to reaching parallel again. This or a mapping like this would fit with one of the haptic mappings we discussed.

• quad-fm-synthesis-out-of-parallel-example.m4v This mapping applies FM Synthesis to a sine wave at 100% if you are furthest from being in parallel and at 0% (just pure sine wave) when you are back in parallel. It also has an unlocking and locking sound when out and in parallel.

[jessegreenberg]

Reviewing https://github.com/phetsims/quadrilateral/issues/4#issuecomment-946949312 with the team:

• For the "Parallels Staccato", @Ashton-Morris suggested that there was less change when the sides were close to parallel. Currently it sounds like they are farther off of parallel than they were.
• @terracoda thought the "Parallels Staccato" was the easiest to understand for a parallelogram. It also was reminiscent of "Blah Blah Blah" :P
• @jbphet thought "Parallels Volume" was the easiest to understand for reaching a parallelogram.
• @BLFiedler thought that the two different tones of "Parallels Staccato" could be confusing. We tried using a single tone for both sides.
• @terracoda and @emily-phet thought that including a success sound would be useful. Just using a representative sound for the shape is not quite enough to indicate whether you are "there" for a parallelogram.
• @emily-phet liked the "Parallels Volume" most of the options so far. "Parallels Staccato" was too much feedback and required a lot of motion since there wasn't a success feedback yet.
• @Ashton-Morris thought the "Parallels Volume" was the most accessible and understandable.
• @jessegreenberg liked the "Parallels Volume" one the most too.
• @terracoda considered that for "Parallels Volume", using volume alone might conflate with other volume changes that come with audio ducking when using speech software in combination with the sim.
  • @emily-phet suggested that volume isn't the only variable in this design so this should still work OK.

We also discussed the new sound designs in https://github.com/phetsims/quadrilateral/issues/4#issuecomment-946949312 and noted they were great continuous representations of the quadrilateral as a shape. @emily-phet suggested we should try a different paradigm. Instead of continuous feedback we try something like "success sounds" coupled with sounds that play when you are in proximity to the goal. The other prototypes are good at conveying shapes but the continuous feedback is not quite helping reach the goal of finding a parallelogram.

We discussed trying out the sound in ratio-and-proportion in quadrilateral. There is a "plucking" sound that happens when going closer/farther to in-proportion. There is also a "success" sound and a moving "in-ratio" sound. @Ashton-Morris said he would review them and demonstrate how they might be used in quadrilateral.

[Ashton-Morris]

Here are the sounds for a different paradigm. Audio files here.

1. One which has a sound for clicking/locking into parallel
2. Unlocking out of parallel
3. And a loop for moving while in parallel

There are 4 different sets to try.

• quad-moving-in-parallel-success-loop-001.mp3
• quad-into-parallel-001.mp3
• quad-out-of-parallel-001.mp3
• quad-moving-in-parallel-success-loop-002.mp3
• quad-into-parallel-002.mp3
• quad-out-of-parallel-002.mp3
• quad-moving-in-parallel-success-loop-003.mp3
• quad-into-parallel-003.mp3
• quad-out-of-parallel-003.mp3
• quad-moving-in-parallel-success-loop-004-rap-choir.mp3
• quad-into-parallel-004.mp3
• quad-out-of-parallel-004.mp3

[Ashton-Morris]

Here is a mockup with the sounds and mapping from Ratio and Proportion synced to the visuals of a quadrilateral. @emily-phet Feel free to tag anyone or we can wait and review it in our next meeting.

https://drive.google.com/file/d/1iyl55HqfO_AmkPwXT0fkb5T38R7etc_m/view?usp=sharing

[jessegreenberg]

The above commit was the first pass at the success sounds.

In the meeting today we wanted to 1) Make it so that the "moving success" sound cannot play when dragging a single vertex. 2) There is a bug where the quad goes in and out of the parallelogram state while dragging a side. That should not be possible. 3) Don't give feedback after pressing the ResetAllButton, sound should be muted when Properties are changing from this.

[jessegreenberg]

See https://github.com/phetsims/quadrilateral/issues/26 for an issue for 2) above.

[jessegreenberg]

The above commits accomplish 1) from https://github.com/phetsims/quadrilateral/issues/4#issuecomment-958085289, as well as 3) for all design paradigms so far. 2) will be more difficult and beyond the scope of this issue, it will be continued in https://github.com/phetsims/quadrilateral/issues/26. THis is on hold until we get to the bottom of that issue.

[brettfiedler]

One quick change prior to interviews:

Let's assign Collection 1 of the Success Sounds as the default sound set on load. Everyone indicated this one was pleasant, if not their favorite.

[jessegreenberg]

Sounds good, "Collection One" is the default now for that paradigm.

[jessegreenberg]

Notes from design meeting on 11/16 -

• There is a delay between hearing the "maintenance" sound. Likely because there is a long tail in the success/failure sounds and the maintenance sound is not allowed to play until these are done.
  • This one needs to be addressed before interviews
• You shouldn't be able to hear the "maintenance" sound if manipulating a vertex. @jbphet demonstrated a case where it was possible to hear this sound when moving a single vertex with a keyboard.
  • This is a bug and should be fixed before interviews.
• The "success" and "failure" sounds do not sound symmetric.
  • This doesn't need to be fixed now, but something to consider as we improve designs after interviews.
• There is a small gap in the audio when the "maintenance" sound is played for a long time. @jbphet suggested that we use a different file format to resolve this.
  • This one needs to be resolved before interviews.
  • Changing from mp3 to .wav may be a fix for this. Also make sure that it starts and ends with a "zero crossing". @Ashton-Morris would be able to verify this. The .wav file is a bit more memory intensive but that may not be a concern for this case. Ill work with @Ashton-Morris to generate a new .wav file.
• There is nothing that indicates that you are maintaining a parallelogram with a keyboard because moving a side will always keep the quad in parallelogram.
  • We discussed changing attributes of the sound based on area of the shape.
  • Something could play at a certain interval of time or distance.
  • No action items now, something to consider in the future.
• Since motion of vertices is velocity based, it is difficult to make reproducible cases. Lets change it to move a certain delta per key press (and as long as it is held down).
  • This needs to be its own issue, it is not necessarily related to sound.
  • See https://github.com/phetsims/quadrilateral/issues/33

[Ashton-Morris]

Adding some sounds.

To resolve "There is a small gap in the audio" This is a seamless loop that should work without gaps or clicks.

• quad-moving-in-parallel-success-loop-001.wav

These two are sounds from other sims that might resolve this JB issue "There is nothing that indicates that you are maintaining a parallelogram with a keyboard because moving a side will always keep the quad in parallelogram."

• quad-parallelogram-with-a-keyboard-indicator-001.mp3
• quad-parallelogram-with-a-keyboard-indicator-002.mp3

[brettfiedler]

Reprioritizing to address the comments in https://github.com/phetsims/quadrilateral/issues/4#issuecomment-970622640 that should be fixed prior to interviews next week.

[jessegreenberg]

There is a delay between hearing the "maintenance" sound.

Fixed in https://github.com/phetsims/quadrilateral/commit/41e456d2f8461a53fdb2ccd6dda989dbd2e32af4.

You shouldn't be able to hear the "maintenance" sound if manipulating a vertex.

Fixed in https://github.com/phetsims/quadrilateral/commit/71c0caf2061bc6f596dd3971b87b1763d926ea0b.

There is a small gap in the audio when the "maintenance" sound is played for a long time

Fixed in https://github.com/phetsims/quadrilateral/commit/d58ab43d3088d482afb7637a250d116917423439

[jessegreenberg]

https://github.com/phetsims/quadrilateral/commit/d58ab43d3088d482afb7637a250d116917423439, I fixed a bug in the "Parallels Volume" paradigm where the sim would crash. But that concludes items required for interviews in https://github.com/phetsims/quadrilateral/issues/4#issuecomment-970622640.

Reassigning back to @BLFiedler.

[brettfiedler]

I believe the above "necessary" issues are fixed from what I can tell. If @jbphet or @Ashton-Morris wanted to give it a try and see if any of the issues still crop up in https://github.com/phetsims/quadrilateral/issues/4#issuecomment-970622640, that would be great but not mandatory.

Dev link for interviews: https://phet-dev.colorado.edu/html/quadrilateral/1.0.0-dev.17/phet/quadrilateral_en_phet.html?showPointerAreas

Let's keep this issue open until after this round of interviews - then we'll carry over any open comments above into the next design iterations.

[brettfiedler]

Closing this issue as the general sound implementation issue in favor of #6. New issues incoming for new sounds as a result of the interviews.