search

ValveSoftware / steam-audio

Steam Audio
https://valvesoftware.github.io/steam-audio/
Apache License 2.0
2.3k stars 162 forks source link

Using Steam Audio spatialization on a UWP device (Hololens) #376

Closed DivinorWieldor closed 4 weeks ago

DivinorWieldor commented 1 month ago

I am in the process of creating an app targeting the Hololens 2, and need the ability to spatialize and simulate realistic sounds from a given source. I need the benefits that come from occlusion, ray tracing, material transitivity, and more. However, it looks like steam audio does not officially support UWP platforms.

Still I would like to see how much of it actually works on the device, and how much of it breaks. To get this, I want to basically force steam audio to run its spatialization algorithms.

Currently, when an audio is spatialized with steam audio, there is not sound. When using the Steam Audio Manager, there is an error stating the platform is not supported. Is there any way to bypass these restrictions?

lakulish commented 1 month ago

@DivinorWieldor While Steam Audio does not currently ship with UWP binaries as part of our release process, the full source code is available, and with slight modifications, it should be possible to get it working on UWP devices. See, for example, the following pull request: https://github.com/ValveSoftware/steam-audio/pull/349.

Briefly, you will have to make changes to the core Steam Audio SDK (phonon.dll), and the plugins for whatever game engine and/or audio middleware you're using. You'll have to modify the CMake scripts to add appropriate compiler flags for compiling in UWP, and potentially update various platform-dependent #define directives at various places in the code.

Since Steam Audio can also build all of its dependencies from source code, it should also be possible to make a minimal set of dependencies run on UWP.

DivinorWieldor commented 4 weeks ago

I would like to note my findings here for anyone in the future looking to run steam-audio on the Hololens.

There is a simple way to force load steam-audio onto the Hololens (UWP), and that is by simply going to Build.cs (Assets\Plugins\SteamAudio\Scripts\Editor) and adding NamedBuildTarget.WindowsStoreApps under the NamedBuildTarget[] supportedPlatforms array. The result of this, however, is that while the HRTF sonification works as expected (similar to Microsoft Spatializer), the reverberations and the results of ray tracing from source do not work. I'm not sure if this requires further edits within the scripts, dependencies, and the underlying code, or whether ambisionic decoding is simply not supported in the Hololens.

A simple workaround to this problem, however, is to run the desired app with holographic remoting. This is similar to developing on VR, where you can run the app on your PC but stream the image to your headset. Holographic remoting supports having the program run under a Windows architecture (you can choose standalone as the build target), while the headset reports all inputs, movements, and interactions to the PC.