search

ircam-ismm / node-web-audio-api

Web Audio API implementation for Node.js
https://www.npmjs.com/package/node-web-audio-api
BSD 3-Clause "New" or "Revised" License
108 stars 14 forks source link

MediaStream Destination Node #91

Open justin0mcateer opened 8 months ago

justin0mcateer commented 8 months ago

You mention the MediaStreams API is somewhat out of scope for this library, but there is an implementation (wrapper) of the MediaStream Source Node. Also, the upstream/wrapped crate appears to implement the MediaStream Destination node: https://github.com/orottier/web-audio-api-rs/blob/main/src/node/media_stream_destination.rs

Aside from perhaps needing to be written manually (vs generated) what are the challenges with creating a MediaStream Desitnation Node wrapper? Is there anyone able to take this on if it were to be sponsored?

b-ma commented 8 months ago

Hey, as said in #27 , the issue is more about what it would mean to have a handle to a stream in a JS environment (meaning that another library would be able to use the handle and underlying stream properly), rather than just providing a wrapper for the upstream object. A possible solution would be consider these just as Node streams (which are not really MediaStreams neither), but then we still need to wait for https://github.com/napi-rs/napi-rs/discussions/1483 . In any case, there is no simple solution right now unfortunately, what would be your use case?

justin0mcateer commented 8 months ago

I don't actually want to handle the MediaStream directly in the JavaScript, I need to hand it off to another library (WebRTC) that is expecting specifically a MediaStream (not a NodeJS Stream).

orottier commented 8 months ago

Hey @justin0mcateer, thanks for bringing this up.

The problem is, like @b-ma states, there is no such thing as the MediaStream in NodeJs. So even if we would provide an implementation here, it will not be compatible with other libraries. (The w3c only specifies the API surface and relation to other APIs such as webaudio and webrtc, not the inner workings). See e.g. https://github.com/node-webrtc/node-webrtc/issues/728#issue-1214108642 for the issues that will arise.

On the web-audio-api-rs side of things I acknowledge this issue, and we solve it by provinding a real-time safe MediaStream shim that allows you to extract the audio samples ( https://docs.rs/web-audio-api/latest/web_audio_api/media_streams/struct.MediaStreamTrack.html#method.iter ) and it allows you to construct an instance from raw audio data ( https://docs.rs/web-audio-api/latest/web_audio_api/media_streams/struct.MediaStreamTrack.html#method.from_iter ).

The NodeJs ecosystem should provide the same so application authors will be able to glue libraries such as node-web-audio-api and node-webrtc manually. So perhaps this issue could be rewritten to "How to extract raw audio without resorting to AudioWorkletProcessor"?

b-ma commented 8 months ago

Thanks @orottier for the further details

I guess the best solution would be to have some compliant C++/Rust/Node implementation of https://www.w3.org/TR/mediacapture-streams/ that everyone agrees to use as middleware and build upon... but this looks like really non-trivial (and a bit like daydreaming for now :)

medemi68 commented 6 months ago

Would it be possible to change the audiocontext destination and convert it to a readable stream? How do we extract a stream from node-web-audio-api without it playing from the speakers of the device? If we can extract the writable stream then in @roamhq/wrtc we could probably use audioSource.onData and pump in the raw chunks.

b-ma commented 6 months ago

Hey, thanks for re-opening the discussion

Would it be possible to change the audiocontext destination and convert it to a readable stream?

I'm really not convinced that's the way to go actually. In my opinion, this question must be solved at the lowest possible level and not on the JS side (maybe it could, but it would probably create some kind of problems that nobody wants to spend time on...)

As @orottier mentioned, on the rust side this is quite opened as MediaStream acts as an iterator that is "a real-time safe MediaStream shim that allows you to extract the audio samples". So if you have some kind of C++ implementation of MediaStream in @roamhq/wrtc, I guess the first step would be make a bridge working at the C++/Rust level (I have personally no idea of how this can be achieved, but happy to help)

Then, once done creating a JS facade over that should be rather straightforward (hopefully)

medemi68 commented 5 months ago

I'm pretty sure that the whole implementation of @roamhq/wrtc is just a wrapper around a C++ API with bindings for Node.