Facilitate Use of Sound Effects.

[Penguin-Guru]

In the meeting today, some of us discussed the importance of (subtle) in-world sound effects and the challenges that currently discourage creators from controlling this dimension. I am not very familiar with the current creation workflow but I wanted to open this issue for discussion.

My view is that there are two logical kinds of sound effects-- constant and triggered. I think there should be a simple menu to attach sound files to entities, and that attached sound files should display check-boxes to enable constant playback of the file. A volume level adjustment would also be very useful here, with values normalised against the interface's default sound effect volume level. Triggered sound effects would need to be referenced from scripts, unless default scripts are supplied for common triggers (e.g. user avatar within X distance, user connects to domain, user performs primary action on entity).

There should probably be a similar way to attach sound effects to zones or even whole domains. These might be used, for example, in the case of sound effects for waves lapping a beach or wind rushing against the user's avatar.

I think it is important for Overte to supply a good sound pack of freely licensed sound effects that creators would be recommended to use for common purposes unless they specifically want something different. This would both simplify the creation experience and encourage a somewhat consistent user experience between domains.

Another long-term idea I think might be useful is adding an app, or a mode to the existing Create app, that would specialise in audio engineering. This would be useful for more technical fine-tuning of audio, perhaps visualising sound waves in-world and allowing for simple customisation of entities' acoustic properties. Simple filtering/editing of the source sound files might be convenient.

I don't know much about how sound effects are currently handled so please discuss. :)

[ksuprynowicz]

Audio zone editing should also be possible from Create App. Right now as far as I know it's only possible by directly entering coordinates in Domain Server configuration website. As for the sound library, it ties very well into an idea of an asset browser, which would have not only sounds, but also other asset types. It could be configured to get them from different sources, like for example local asset server, or external online repositories with open licenses. Such thing would make virtual world creation a lot easier.

[HifiExperiments]

I think we should treat sound effects as normal objects in space. We should create a Sound Entity type with properties like url, volume, positional or not, drop off, repeat or not, etc. We should have methods to pause/play/restart sounds that can be triggered on the entity. Internally, this entity would just manage the existing Sound objects.

For audio zones, we’ve talked about this for a while and it’s totally possible, but I’m less familiar with the other mixers so I don’t know how to configure what gets sent to each one. I know we already send zones to the avatar mixer for Hero zones. We just need to duplicate that logic to also send zone data (specifically, just their transforms + audio properties) to the audio mixer. And then hook that up to the existing audio zone code there.

[ksuprynowicz]

It's a good idea. This way it would be very easy to integrate with create app. By the way, I'm wondering if there's a way to make Interface support multiple protocol versions? It would allow a lot easier experimentation.

[Penguin-Guru]

I don't know how this is currently handled or if it is remotely practical but I like the idea of sound volume decay and frequency attenuation being applied by the environment. The user experience will probably be best if sound volume decay is consistent, at least in a zone or domain, and allowing the sound profile to be affected by the shape and materials in the environment would make the experience more dynamic and immersive. For example, increasing distance from a sound object should reduce the perceived volume, walking behind an obstruction should also reduce the perceived volume and perhaps change the perceived directionality, and materials used in the environment could have different (ideally frequency specific) absorption and reflection profiles.

[HifiExperiments]

RE: protocols, people have proposed some solutions but I’m extremely skeptical. all would require some changes to the networking code. I simply don’t think it’s viable to have multiple interfaces connecting to a domain that disagree on how to handle properties. Any property that affects rendering or physics would be essential to a domain experience…you don’t want people seeing different things or disagreeing on how to simulate physics. Perhaps it could be allowed just for development, but I think it’s an extremely dangerous area.

RE: realistic sound propagation…unsurprisingly it’s extremely hard to simulate proper sound, never mind in real-time at the low latency you need for sound. I think we’re pretty far out from that being viable. But on a basic level you could sorta fake it with some very tricky scripting logic, maybe? I think the existing audio zones do some form of this but only interacting with the edges of the audio zone (e.g. big open rooms can “sound” big), not the objects in the zone.

[Penguin-Guru]

Yes, I don't expect truly realistic sound propagation is an short-term option but I wonder if sound effects or the related U.X. could be handled in such a way that it would be easier to adopt in the future. It seems like something that could be very easily tuned for performance if it were supported-- maybe not. I think designing the U.X. to approximate real physical behaviour would be intuitive for content creators.

[HifiExperiments]

the PRs for both Sound entities and Zone Audio properties have been merged, but I'm going to leave this open because there's some other good ideas in the original issue