search

BabylonJS / BabylonNative

Build cross-platform native applications with the power of the Babylon.js JavaScript framework
MIT License
766 stars 132 forks source link

Sound support #31

Open CedricGuillemet opened 5 years ago

CedricGuillemet commented 5 years ago

crossplatform audio engine https://github.com/jarikomppa/soloud Do you know any other one?

bghgary commented 5 years ago

https://github.com/resonance-audio/resonance-audio

CedricGuillemet commented 5 years ago

Some more tiny libraries here https://github.com/nothings/single_file_libs#audio

bghgary commented 3 years ago

From forum post: https://forum.babylonjs.com/t/babylonnative-questions/9716

https://github.com/LabSound/LabSound

chrisfromwork commented 3 years ago

some real time filter libraries:

https://github.com/vinniefalco/DSPFilters https://github.com/berndporr/iir1

chrisfromwork commented 3 years ago

https://github.com/andrewrk/libsoundio https://github.com/PortAudio/portaudio

chrisfromwork commented 3 years ago

https://github.com/cycfi/q

Drigax commented 3 years ago

We may also want to consider: https://www.videolan.org/vlc/libvlc.html

It may be overkill for our needs, (we also may want to consider the implications of its license) but seems to have robust support for 3d sound, media streaming, and filter effects if we choose to implement a subset of WebAudio.

CedricGuillemet commented 3 years ago

@Drigax Looks interesting for video playback support as well. I'm curious about the footprint on mobile.

Drigax commented 3 years ago
Project Name Link To Project Does it implement an audio backend or directly interact with OS-included backends? Does it implement DSP Effects? Does it use a modular or node-based processing pipeline? What does it do? Windows/Win32 Windows/UWP Android MacOS iOS Linux License Notes
Chromium https://source.chromium.org/chromium/chromium/src ✅ ✅ ✅ Full browser implementation. Includes implementations for WebAudio API with platform audio backends ✅ ✅ ✅ ✅ ✅ ✅   This is our gold standard (because it pretty much is the standard.) Supports all the funtionality of WebAudio without additional dependencies, and works as a full multimedia backend.
SoLoud https://github.com/jarikomppa/soloud ❌ ✅ ❌ Portable audio processing library with extensible backend support ❔ ❔ ❔ ❔ ❔ ❔ Zlib This isn't necessarily an all-in-one solution, as it relies on pulling in an audio backend as a dependency. However it has a number of audio effects implemented to support the subset of WebAudio that we need to implement.
libSDL https://github.com/libsdl-org/SDL ✅ ❌ ❌ Low level cross-platform I/O interface ✅ ❔ ✅ ✅ ✅ ✅ Zlib This is more than just a cross platform audio backend (and interface to other desired backends), this library aims to solve multiple cross platform I/O issues such as input/haptics, graphics, sensors, file system access, etc…
Resonance Audio https://github.com/resonance-audio/resonance-audio ❌ ✅ ✅ Audio processing library with node graph and a wide library of built-in DSP effects ❔ ❔ ❔ ❔ ❔ ❔ Apache 2 This is a DSP middleware package. While it should support the functionality of WebAudio, it requires pulling in an audio backend implementation
JACK https://github.com/jackaudio/jack2 ❌ ❌ ❌ Audio server that allows for inter-process audio routing. ❔ ❔ ❔ ❔ ❔ ❔   This isn't an audio processing engine, or backend. JACK is an audio server that allows audio to be sent between processes ex: sending audio from one DAW to another that doesn't explicitly support each other. This doesn't seem useful for this usecase, but this is useful to list so its not misconstrued.
MiniAudio https://github.com/mackron/miniaudio#supported-platforms ✅ ⚠️ ❌ Single-file crossplatform audio I/O ✅ ❔ ✅ ✅ ✅ ✅ Public Domain OR MIT This is a minimalist, single file audio engine with built in support for OS-shipped backends (DirectSound, WASAPI, CoreAudio, ALSA, Aaudio, etc…) There are some DSP effects included as well.
libsoundio https://github.com/andrewrk/libsoundio ✅ ❌ ❌ Cross platform audio I/O library ✅ ❔ ❌ ✅ ❔ ✅ MIT This is a cross platform audio I/O library that interacts with a number of platform and 3rd party audio backends.
portaudio https://github.com/PortAudio/portaudio ✅ ❌ ❌ Cross platform audio I/O library ✅ ✅ ❌ ✅ ❔ ✅ Custom, Permissive License This is a cross platform audio I/O library that interacts with a number of platform and 3rd party audio backends.
libvlc https://wiki.videolan.org/LibVLC/ ✅ ✅ ✅ Cross platform media I/O library ✅ ✅ ✅ ✅ ✅ ✅ LGPLv2 This is a cross platform media player, with support for network streaming of media, as well as local playback. Support for simultaneous playing of one-shot sounds are not known, given the nature of the full application. This does have directly plug into platform audio and video backends.
LabSound https://github.com/LabSound/LabSound ✅ ✅ ✅ WebAudio Port that uses libnyquist for DSP and miniaudio as a backend ✅ ❔ ✅ ✅ ✅ ✅ BSD This project began as a port of WebKit's WebAudio implementation. Some 3rd party solutions are used for audio processing and backend, namely libnyquist and miniaudio.
                           
                           
Key:                          
❔ Not conclusive, need more information                        
✅ supported /included                        
❌ Not supported / not included                        
⚠️ Some support/some inclusion                        
Drigax commented 3 years ago

I've spent the past few days researching various popular audio solutions, and have tabulated some of their qualities into an easy to digest chart. This should allow us to compare solutions/a set of solutions to see what direction we want to go with regarding incorporating existing audio projects. The chart above will continue to be populated as I look at more projects.

Drigax commented 3 years ago

Currently we are still investigating the feasibility of incorporating Chromium build artifacts that contain WebAudio, and the Chrome media system libraries and dependencies. Ashu Tatake has taken the first steps in this approach.

@ryantrem had a very interesting concern regarding this first option, Apple has been known to inspect and remove apps that contain Chromium source. We should look into which apps were known to be removed for this reason, if there was any justification provided, and whether Apple has a clear policy on whether it is a blanket ban on any chromium-derived binaries, or specific parts of Chromium which are not allowed.

I'll follow up by investigating this as well.

EDIT: I'll leave this comment up, in case this becomes a future concern again, so far I can only see this issue with Electron as related: https://github.com/electron/electron/issues/20027

since Electron was using Apple internal APIs, violating the TOS. Though we should make sure there are no similar problems with leveraging the Chromium source as well.

Drigax commented 3 years ago

I've written a blog post here, outlining the plan moving forward for this feature: https://babylonjs.medium.com/audio-in-babylon-native-charting-the-way-c6bc6d5c0d74

Put shortly, it seems best to start with the most capable single solution and explore its feasibility before looking at combinations+custom solutions.

Starting with existing implementations in browsers, Chromium and Firefox's implementation of WebAudio come with too much additional work to get the codebases in a shape where integration would be trivial, as well as questionable build processes.

I am now investigating the feasibility of LabSound, another WebAudio implementation that appears to be "standalone" in its build process and consumption. A large plus is that the project was created to use CMake as its build system, which suggest that it should not be too difficult to integrate into BabylonNative's build system

Drigax commented 3 years ago

Update 2021-05-17

The project and samples were successfully built on Windows, it may be worth taking the next step and getting the project to build as a part of BabylonNative's build.

I'll have to look into how to isolate building the engine+dependencies as a consumable lib without building the additional samples

Drigax commented 3 years ago

Update 2021-06-07

My work is currently staged on: https://github.com/Drigax/BabylonNative/tree/add-labsound (BabylonNative) https://github.com/Drigax/Babylon.js/tree/add-native-audio (Babylon.js)

The native changes currently implement a Plugin component for NativeAudio, and modifies the Win32 playground to initialize NativeAudio. Currently the NativeAudio plugin intializes N-Api wrappers for the WebAudio classes needed to support Babylon.js's Sound and AudioEngine classes.

In Babylon.js some changes were made to engine instantiation so that the NativeEngine can have its AudioEngine initialized using the Engine constructor, also the AudioEngine was copied and modified to be usable by the NativeEngine (removing all dom-related functionality) as the NativeAudioEngine implementation.

Drigax commented 3 years ago

Moving forward there are some outstanding work to be done until I believe this work is ready for merging:

bghgary commented 3 years ago

As noted here (https://forum.babylonjs.com/t/source-for-n-api-implemented-bindings-of-web-audio-found/22132), might be interesting to check out this also:

https://github.com/node-3d/webaudio-raub

FranticOwl commented 1 year ago

@bghgary, more than a year later and this is still an open issue?

Is there any actual plan to build support for Babylon Native, so that frameworks such as React-Native can simply use the sound API from Babylon to add audio to scenes?

bghgary commented 1 year ago

Unfortunately, sound support is not actively being worked on right now. We attempted to start this a couple of times, but they both didn't work out for various reasons. cc @docEdub

rootVIII commented 1 year ago

Would love to see audio functionality added to this project.. even if only for a few select targets/OSs.

markgrossnickle commented 11 months ago

No ability to play audio seems like a deal breaker for many game projects. It's why were are going to go with another platform fwiw.

matthargett commented 8 months ago

As we approach the next phase of our product, which includes bringing Babylon React Native to visionOS, Spatial Audio is a need that we have. What can we do at Rebecker Specialties to help close this key gap that also affects using Babylon React Native on AOSP-based headsets (Pico 4, HTC Vive XR/Focus, etc)?

ryantrem commented 7 months ago

Love the idea of having community help getting this implemented! @bghgary thoughts on this? Could @matthargett pick up where Nick left off?

bghgary commented 7 months ago

Let's check with @docEdub. He was looking at this at some point. Any thoughts @docEdub?

CedricGuillemet commented 7 months ago

Valve Steam audio supports major platforms and is apache 2.0 https://github.com/ValveSoftware/steam-audio

docEdub commented 7 months ago

Love the idea of having community help getting this implemented! @bghgary thoughts on this? Could @matthargett pick up where Nick left off?

I was able to hack together a BabylonNative audio implementation last year based on Nick's work. Here's the blog post I wrote afterward: https://medium.com/@babylonjs/moving-the-ball-forward-on-babylonnative-audio-44d54de019bc.