Sonification Audio Tool Concept

[Ashton-Morris]

In a recent meeting with Emily and John we talked about expanding the structure of the Sound Options Dialogue (which can be seen here, click PhET and then options on bottom left) which allowed for real-time sound modifications and adjustments.

This seemed like a great tool to consider working on, and it would allow us to iterate on the sound design faster and to more easily collaborate and communicate with other team members or learners about the audio. It would increase the fluency of audio terminology and possibilities of audio within in the sim's, and would also be a great tool when working with students and doing co-design. Below is my mockup of how it could look and operate.

I have also created a sound design document with further explanations and videos showing how the sound tool might operate which can be found here.

@terracoda , @emily-phet asked me to do my best writing short descriptions of how and what each component does. I added some at the bottom of the document. She felt that this would a great tool to have descriptions on.

I feel that it should have the most universally understood language concerning the audio as to benefit the most people regardless of their experience with audio. Ex. Volume is better known than, Output or Level.

Since this was not something that was brought up during our Rocks meeting and an idea that I brought up on the fly during our last meeting, I am not sure when this will be worked on or completed. But I wanted to give this idea a space to live and a place to have discussions about it.

[jbphet]

I'm going to mark this as deferred for now, and will work with @emily-phet to prioritize the refinement and implementation of this idea once she's back. I have a reminder on my calendar in May 2021 to revisit.

[Ashton-Morris]

@emily-phet This is the issues about the sound tool to check out.

[emily-phet]

Reviewed this at @Ashton-Morris 's request. Thanks @Ashton-Morris for all the detailed documentation (and description language) of your idea. From my recollection of when we spoke about this last in sonification meeting, some of the proposed features are not currently possible within the scope of Tambo, and some may potentially not be possible.

In tomorrow's meeting, we'll probably just naturally go through the progress and expected timelines for the current sound projects (BASE and GHE). I'd like to check in if there are any more on the horizon that perhaps KP has brought up. From there it'll be more clear when @jbphet will be able to fit efforts on this feature into his "queue". Probably he'd start with thinking through what you have here, what's currently possible, what's not, etc., and doing some back and forth iteration with you and (and possibly me and others) to make a clear spec for himself.

I think @BLFiedler may have used with the kids in his Friday classes the sound menu for BASE, and he might have some ideas to contribute as well.

[brettfiedler]

I think my biggest general comment is that since we have used these types of things a few times with classrooms, it'd be good to make the variables/settings as plain language as possible, which it looks like there was a good pass done on these so far. The guide above would be necessary, but there is probably a bit more tweaking we could do?

I'll wrap back around for some more specific thoughts.

[Ashton-Morris]

I showcased the audio tool we used during the development of BASE with the IDRC during the Design Crits. meeting. Someone else brought up language and the challenge of creating plain language that all learners can understand and conceptualize concerning the audio. We did come up with a good solution to try when we circle back to this issue:

• We should attempt to have the audio play back automatically after a learner has adjusted a parameter. Ex. You slide the pitch bar all the way to the right. As soon as you let go of your mouse or keyboard from that interaction the sound automatically plays that the new changed pitch. We felt that quick experimentation and iteration on the part of students would teach them how the sounds changes with each section much quicker than we could teach them by explaining it with text.