Create a soundscape for the observation window

[jbphet]

The sound design team has been talking about creating a "soundscape" with multiple layers of sounds to portray/compliment the things that happen in the observation windows on the waves and photons screens (we'll worry about the layers screen later). This issue is where we will track the requirements and ideas.

[jbphet]

The things that we want to add sound for are:

• ground temperature
• photon absorption and re-emission on the photons screen
• wave absorption and re-emission on the waves screen
• possibly something unique in the background of each time period when in "time period selection" mode

[jbphet]

@Ashton-Morris demoed some pretty cool demos at this week's sound design meeting, so it's probably time to start trying to put some sounds into the sim. Can I get the background drone, wave emission, and wave reflection sounds from the demo?

[Ashton-Morris]

Here are some base sounds to start with.

• waves-wave-emission.mp3
• waves-wave-reflection.mp3
• greenhouse-temperature.wav

[Ashton-Morris]

Also, @emily-phet wanted me to export a short mockup video I showed the team of the photons screen. Here it is.

[emily-phet]

Thanks @Ashton-Morris !

[Ashton-Morris]

During yesterdays meeting I showed a mockup of how the audio for the temperature would work. We decided that it was a good starting direction and that we should attempt to replicate it in sim. Here's how the audio functioned in the mockup:

We have 3-4 audio files that fade into one another. Starting from low temperature all the way up to the highest temperature, this works by using a crossfade between files and having that be modulated by the temperature. The audio files will be a single note ambiance each being the same note.

Emily suggested to have a 4th ambient file noting the temperature zone that human life flourishes comfortably, this will be a slightly positive sounding ambient sound.

At the end of that signal chain we will have a filter where the cutoff frequency follows and is modulated by the temperature. We decided to test it with a Bandpass filter and a Lowpass Filter in the sim.

This is an rough example of how it looks in my software.

[Ashton-Morris]

I am adding 4 ambiences to try when @jbphet get's the time to start testing our temperature concept.

• greenhouse-temperature-low.wav
• greenhouse-temperature-high-003.wav
• greenhouse-temperature-medium-human-ideal.wav
• greenhouse-temperature-medium.wav

[jbphet]

This was discussed in the 6/29/2021 meeting, and the next step is to try integrating all of these sounds and cross fading between them at the various temperature values, something like the diagram above. We'll try the filter idea after that.

@Ashton-Morris - Can I get mp3 versions of these sounds? I'm a little concerned about how much size this will add to the sim with 4 uncompressed loops.

[Ashton-Morris]

Sure I have added the files here.

• greenhouse-temperature-low.mp3
• greenhouse-temperature-medium.mp3
• greenhouse-temperature-medium-human-ideal.mp3
• greenhouse-temperature-high.mp3

[Ashton-Morris]

We reviewed the progress that JB made this past week and we were all impressed. He added all of the ambiences in the above comment and added the fades between them.

Overall the fades and transitions and transition time when going through temperature changes sounded very cohesive and pleasant. The notes on what could be improved and what to do next are:

Todo Ashton

• [x] Create two alternatives to the low, medium, and high ambiences for us to compare. (We felt that the low had a lot of movement, and that the high was "too crispy")
• [x] Create a variation of the "medium-human-ideal" ambience. (In context we all felt that the choir "Aahh" sound was too "in-your-face")
• [x] Check the pitch of the ambiences in contrast with the "Start Sunshine" sound. (The ambiences seemed to create some dissonance and sounded like they were one semitone off)
• [x] Make sure all of the ambiences have a short fade in and fade out at the beginning and end. (We were hearing a pop during the loop)
• [x] Create a few Wave Emission, and Wave Reflection sounds for us to test out alongside the ambiences.

@jbphet

• [x] Create a slow fade in to the first ambience when the temperature begins to register at the start of the sim.

[Ashton-Morris]

Adding these assets now.

• greenhouse-temperature-low-v2.mp3
• greenhouse-temperature-medium-v2.mp3
• greenhouse-temperature-medium-human-ideal-v2.mp3
• greenhouse-temperature-high-v2.mp3
• greenhouse-temperature-low-v3.mp3
• greenhouse-temperature-medium-v3.mp3
• greenhouse-temperature-medium-human-ideal-v3.mp3
• greenhouse-temperature-high-v3.mp3
• greenhouse-wave-emission.mp3
• greenhouse-wave-emission-vibrato.mp3

[Ashton-Morris]

In todays meeting we mixed and matched a few of the above ambiences to create a nice blend.

// favorite set from the 7/13/2021 sound design meeting (was a mix-and-match sort of thing)
temperatureLowV3Sound,
temperatureMediumV2Sound,
temperatureMediumHumanIdealV3Sound,
temperatureHighV2Sound

JB showed a few pieces of music with nice ambiences in the background and we decided to create another set with those as inspiration. The new set would have a chord or multiple notes in the ambience to give it a little more movement and a lot less ear fatigue.

AM Will work on creating that new set of ambiences next for next week.

JB Will continue working on adding the wave emission and reflection sounds so we can hear them in context with the ambiences.

[Ashton-Morris]

I added these sounds which I had intended to upload in the above comment.

• greenhouse-wave-reflection.mp3
• greenhouse-wave-reflection-vibrato.mp3

[Ashton-Morris]

I created 4 harmonious ambiences for us to test out. They all have the same base note and differing secondary notes.

• greenhouse-temperature-rising-with-base-note-low.mp3 (first and a second)
• greenhouse-temperature-rising-with-base-note-medium.mp3 (first and a fourth)
• greenhouse-temperature-rising-with-base-note-human-ideal.mp3 (first and a fifth)
• greenhouse-temperature-rising-with-base-note-high.mp3 (first and a seventh)

[Ashton-Morris]

@emily-phet Here is an example of how the above Harmonious Ambiences would sound with random IR sounds in the pentatonic scale playing also.

The white line going across the screen is the temperature.

• Harmonious_Ambience_Example.mp4

[emily-phet]

@Ashton-Morris I like the basic feel of it!

• I'm guessing they'll be refinements to the timbre, etc to the photon sounds, but the basic idea sounds good to me.
• We'll need to clearly be able to distinguish sounds that indicate certain events. This example sounds very random to me, like I imagine it might be hard to distinguish absorption from re-emission events (though who knows until we try it, it's always a different experience when you have the visual + sound). My guess is we'll want a set of very very similar sound variants for absorption, and another set of very very similar sounds for re-emission, and each set is clearly distinct from the other.
• I have a hard time imagining how it would work if the background temperature sound was more dynamic (as we discussed last week), but still open to trying it and seeing how that variant goes.

Our upcoming meeting this week will probably be a good time to talk through and check-in on our current mappings between sounds and photon / wave events, so these ideas mesh with the latest understanding of everyone, and we're all on the same page.

[Ashton-Morris]

@jbphet Here are the harmonious ambiences with a type of rhythm to them for you to test out.

• greenhouse-temperature-rising-with-base-note-and-rhythm-high.mp3
• greenhouse-temperature-rising-with-base-note-and-rhythm-medium.mp3
• greenhouse-temperature-rising-with-base-note-and-rhythm-human-ideal.mp3
• greenhouse-temperature-rising-with-base-note-and-rhythm-low.mp3

[Ashton-Morris]

I have come cool mockups to show in tomorrows meeting but I will post them here as well so they can be heard outside of your meeting.

greenhouse-effect-waves-continuous-ir-with-temp-blips-mockup.m4v

• This has a warm tone for the IR being emitted from the ground
• A organ sounding tone for the IR being absorbed and re-emitted in the atmosphere
• A blip that goes up in pitch as the temperature readout changes for the IR being absorbed into the ground. (This version has no Ambient Temperature Sound) I couldn't make that sound work because it seemed to be too many different things that one wave needed to do.
• An echoey sound for the IR being lost to space
• It has a staccato sound leading into a continuous sound for each interaction. greenhouse-effect-waves-discrete-ir-with-temp-blips-mockup.m4v greenhouse-effect-waves-discrete-ir-without-temp-blips-mockup.m4v

These have the the same sounds as the continuous one but they are short and not continuous. One has the ambient temperature sound that we worked on with some blips that go along with the temperature and one has it without the blips.

@emily-phet take a listen!

[jbphet]

Some early interviews were done on the soundscape, specifically the ambient-ish sound that represents temperature, see https://github.com/phetsims/greenhouse-effect/issues/55. We discussed the outcome of these interviews in the 8/3/2021 sound design meeting, and decided that we'd like to pursue something that was possibly a more overt sonic representation of the temperature. Here are some things that we

• Using an LFO on a single loop that increases in frequency as the temperature goes up. We could try both a sinusoidal LFO and a square, randomized sound change as well. @Ashton-Morris mentioned the term "sample and hold (S&H)" when we were discussing this
• Changing the playback rate of a single loop as the temperature changes
• Cross-fading between 3 octaves

We will start this process using the loops that we already have and are using for the cross-fade examples.

@Ashton-Morris will mock these up before we try to put any of them into code.

[Ashton-Morris]

During yesterdays meeting I had the time to create some of these mockups and go through them with the team. We found some directions that might work and found out which directions probably wouldn't work.

We decided that @jbphet would try 3 approaches.

1. The temperature is tied to a low pass filter cutoff frequency
2. The temperature is tied to a band pass filter cutoff frequency
3. The temperature is tied to a the playback rate of the ambience making it go up and down in pitch as the temperature changes

Here is the 4 octave ambience audio file for 1 and 2. greenhouse-effect-temperature-base-ambience-4-octaves.mp3

And I think you should use this file for 3. greenhouse-temperature-rising-with-base-note-low.mp3

Additionally I decided to mockup all of the variations incase we need to re-review them or reference them. @emily-phet if you're interested. Individual mockups can be found below or a zip file with all mockups can be downloaded here.

• greenhouse-effect-waves-temperature-band-pass-filter.m4v
• greenhouse-effect-waves-temperature-band-pass-filter-w-blips.m4v
• greenhouse-effect-waves-temperature-low-pass-filter-w-blips.m4v
• greenhouse-effect-waves-temperature-low-pass-filter.m4v
• greenhouse-effect-waves-temperature-with-abience-3-octave-change.m4v
• greenhouse-effect-waves-temperature-with-abience-volume-tempo-change.m4v
• greenhouse-effect-waves-temperature-with-abience-pitch-change-mockup.m4v
• greenhouse-effect-waves-temperature-with-abience-pitch-change-mockup-w-blips.m4v

[Ashton-Morris]

During our August 17th Sonification Meeting Emily and I thought it would be best to try this approach for the IR ReEmission in sim.

@jbphet Please view this mockup greenhouse-effect-waves-ir-reemission-and-temperature-mockup In the above video the sound plays each time the IR ReEmission occurs in the air. In the background we have the Temperature ambience with a Low Pass Filter.

• Here is the sound greenhouse-effect-ir-reemission

[jbphet]

The first version of the IR re-emission sound was demonstrated in today's sound design meeting and, at least so far, we like it.

[jbphet]

The soundscape was turned off for an initial early delivery of this sim, but we want to be able to continue to refine it, so I've added a query parameter to turn it on. It is soundscape=true.

[jbphet]

I've added options to control the temperature sound and whether IR re-emission triggers a sound. The next step is to implement the sound generators that go with it and hook it all up.

[jbphet]

The sound generators have now been created and hooked up to the "Options" dialog.

@Ashton-Morris and @emily-phet - Please take this version for a test drive (link below) and log your thoughts on what works and what doesn't with respect to temperature and IR-reemission sounds. Keep in mind that we have a lot of flexibility here in terms of what loops and frequency ranges we use, as well as how they map to the temperature.

I think you'll enjoy trying this out. It's pretty cool.

Link: https://phet-dev.colorado.edu/html/greenhouse-effect/1.1.0-dev.1/phet/greenhouse-effect_en_phet.html?soundscape=true

[Ashton-Morris]

Wow these do sound cool!

• Multiple loops with cross-fades: This sounds the most musical and perhaps pleasant but I am not feeling like the the temperature is changing when I am hearing it, I am just feeling like the audio is changing.
• Low Pass Filter: This sounds great! I’d like to test it with more sharper resonance on the Q of the filter. Just a little more. The fullness on any temperature works well.
• Band Pass Filter: It sounds too thin sounding and I am not feeling like its getting hotter in temperature with the thinness on the max temperature.
• Playback rate change: Sounds really cool isolated, but I think it it’ll get confused with a vehicle slowing down and speeding up.

These all came out well. My vote if for the one option 2: Low Pass Filter

[Ashton-Morris]

EM Said we should move forward with the Low Pass Filter option and work improving that when it comes to the Q.

[jbphet]

EM Said we should move forward with the Low Pass Filter option and work improving that when it comes to the Q.

We experimented with this a little during today's sound design meeting, but I'll make it easier and more definitive for the next meeting to that we can determine the value we want to commit to. We're mostly concentrating on the lowpass filter, but would like to try messing around with the Q on the bandpass as well.

[Ashton-Morris]

We decided today to try the Waves screen with continuous sounds for the IR coming up from the ground and the IR ReEmission. We also will try it with discrete sounds that play at the start of the loop and without. I am adding these files to be put in.

• greenhouse-effect-waves-ir-reemission-loop.mp3
• greenhouse-effect-waves-ir-loop.mp3
• greenhouse-effect-waves-ir-starting-sound.mp3
• greenhouse-effect-waves-ir-reemission-starting-sound.mp3

In total we will have three versions to compare (click to view mockup):

1. Temperature + Discrete IR ReEmission sounds
2. Temperature + Looping IR and IR ReEmission sounds
3. Temperature + Looping and discrete IR and IR ReEmission sounds

[jbphet]

During today's sound design meeting, we clarified a few points about the experiment described in the previous comment:

• Each of the waves coming from the ground or the atmosphere should have its own loop, so for instance, there could be three instances of the greenhouse-effect-waves-ir-loop sound playing at once.
• When a wave interacts with the atmosphere and starts coming back to the surface, the loop for that wave coming from the ground stops.
• For now, we shouldn't worry about making the volume of the loops relate to the intensity of the waves, and should just use a single level. We might revisit this later if we decide to continue with this "layered loops" approach.

The code for this is partially written, so I'll get it working with these points in mind and then will assign to the sound design team members for review.

[jbphet]

@emily-phet and @Ashton-Morris - I have the IR discrete sounds and loops working, please try them out in https://phet-dev.colorado.edu/html/greenhouse-effect/1.1.0-dev.2/phet/greenhouse-effect_en_phet.html?soundscape=true and record your thoughts here.

I played around with it for a bit, and found it very instructive to switch around what was on and off through the options dialog, but it definitely felt like too much when all of the sounds were on. I'll be interested to hear your take.

[jbphet]

The sound design team reviewed the latest version with the IR sound options in today's sound design meeting, and here are what we're thinking should be the next steps:

• [x] Change the IR from atmosphere loop to only play when interactions are occurring with the atmosphere, as opposed to when IR waves that originated in the atmosphere exist
• [x] Map the volume of the loops to the intensities of the waves, but have option to enable/disable this feature so that we can hear the sounds in context without any changes in volume
• [x] Make the ground wave loops independent of whether atmospheric interactions are occurring

[jbphet]

We discussed the latest soundscape progress this week at the sound design meeting, and feel like this is progressing reasonably well. The next step is to turn down the temperature sounds a bit, then review the following configuration with the larger design team:

I'll request design time for this, since our usual weekly design meeting has been canceled while I'm focusing on the conversion to TypeScript.

[jbphet]

Another thing to discuss with the larger design team: If we keep the one-shot sound that signals the beginning of an IR-atmosphere interaction, should we force them to be staggered?

[jbphet]

@arouinfar, @kathy-phet, @Ashton-Morris and I just met to review the current implementation, and the conclusion was that with the settings shown in the comment just above the sound design is in a state where @arouinfar can start doing some interviews on it. I asked about whether to stagger the "IR interaction started sounds", and the feedback was that no, it's not really worth spending the time on it. We should do the interviews on what we have, get the other screens working, and then refine this based on what we learn.

The next steps are:

• [x] @Ashton-Morris will have a listen and see if he feels like we should adjust the relative volume of the temperature background sound and the IR-wave-specific sounds
• [x] @jbphet will then create a dev version that uses the identified configuration as the defaults and pass it along to @arouinfar for interviews

Assigning to @Ashton-Morris for the first item above.

[Ashton-Morris]

I couldn't get that sim to load from phettest. I had an unknown TypeError.

But from the last dev version and our last meeting I can recall thinking:

1. Turn down the temperature sound 10%
2. Turn up the IR Emission and Waves sounds 10%

I think those changes should suffice for the interviews.

[jbphet]

The soundscape has been in a state to support interviews on the "Waves" screen for a while, though I don't think they've happened. I think it's time to close this issue and do any other work, such as refinements to support other screens and modifications due to feedback from interviews, in other GitHub issues. Closing.