search

alnitak / flutter_soloud

Flutter low-level audio plugin using SoLoud C++ library and FFI
MIT License
217 stars 22 forks source link

feat: use PCM audio data to build a sound #144

Open alnitak opened 1 day ago

alnitak commented 1 day ago

Description

Would be nice to be able to send PCM data to the player and have the player be able to render it. This could be done by extending the SoLoud C++ AudioSource class to have a BufferStream class that can be used as a normal AudioSource sound.

The advantage of this is that the player can be used as a normal [AudioSource] and can play audio data that could be streamed from the web or created directly from Flutter. This new kind of [AudioSource] can then be able to use all the features of the flutter_soloud like filters, faders, oscillators and so on. Think for example of some AI web services that let you compose voices like this.

Additional Context

Would also be nice to have a way to send PCM data directly to the output device without creating a new [AudioSource]: compose some audio PCM in Flutter and send to the player output. But this is maybe for another feature.

alnitak commented 1 day ago

Experimenting with the BufferStream I came up with some good results, but opinions and suggestions are welcome.

What I have done so far is to simulate a WebSocket connection that streams PCM data using ffmpeg. Here the source code. The only needs to use it, are to have ffmpeg and websocketd installed, and in the main.dart modify final audioPath to point to an audio file on your PC. websocketd Here you choose in which format you want to send the PCM audio data.

In the websocket branch there is an example that demonstrates this feature in `lib/buffer_stream/websocket.dart' file. https://github.com/user-attachments/assets/e7c443ef-7c06-410a-91a4-804b698a328f

lib/buffer_stream/generate.dart is an example of how to generate PCM audio data within an Isolate and send them to the AudioBuffer.

I have tested on Linux, Android, and Windows. It should work also on Mac and iOS, if you feel good please try on those. No web support ATM.

cc: maybe @maks could be interested in this?

maks commented 20 hours ago

@alnitak thank so much for tagging me on this as yes I'm very much interested!

I've quickly glanced at this so far but not sure it looks like exactly what I would like to be able to do: generate and playback PCM data in realtime. But I think maybe I'm missing something here: in the example you are pre-generating the data into a in-memory buffer (the for loop in generateWailingSirenSoundPCM()) as I don't see any callback that would be coming from the native audio system code that requests the audio data on the fly (as it requires it for pushing out to the OS's sound output API)? For me thats the main thing: I want to be able to make musical instruments so need to generate the audio data in realtime with low latency, ie. as the user plays notes via for example a MIDI controller input.