padenot
commented
2 years ago This repository is home to the Web Codecs specification, and this issue tracker is generally not for support questions.
Questions on how to do something with the Web Codecs are better suited to Stack Overflow or other locations for support. I'll add an agenda item to figure out what / where to do support for the next call, and in the meantime, I'll subscribe to this webcodecs Stack Overflow tag to try to build momentum there, there's not a lot of activity and knowledge is scattered.
That said, what you want to do is to use an AudioWorkletProcessor and a wait-free ring buffer to extract the audio from your source, and to send it to a worker for further processing (in your case encoding via web codec, muxing, stream to disk). https://ringbuf-js.netlify.app/ has an example that does more or less that: capture an arbitrary audio source to a worker, "mux" and "encode" into a WAV file (for simplicity, as this isn't the purpose of the example). It offers a file to download instead of streaming to disk, but that's not the hard part. With this setup, there is some main thread setup, but no main thread activity after the setup (in fact, the example artificially loads the main thread to make this clear).
The video side you seem to have covered. I'm not sure that we have good, generic muxers to recommend, but this is also something the editors and other participants to the W3C Media WG are trying to figure out.
Is there a sample that can encode a ReadableStream transferred to a worker into a webm format with audio and stream it to the file system using the new File System Acccess API? I was able to use one of the samples which has a modified webm-writer2.js file and achieve everything video related on a worker, but I am stuck on finding an encoder and muxer that can do both audio and video streams in the worker using the new WebCodecs API.
This example has a workaround for audio, but it relies on AudioContext which can only be used on the main thread. This can become problematic and cause interruptions on the main loop as it needs to constantly read chunks from an event. https://github.com/xiph/opus/issues/221
My goal is to record a video/audio stream, transfer the MediaStream entirely off the main thread for processing and stream, mux it all into a single webm container, and save it to the file system without any memory limits. Does anyone have a working solution that can build off that sample and include audio?
Here's some references:
webm-writer2.js (video only so far) https://github.com/w3c/webcodecs/tree/main/samples/capture-to-file
Audio encoding problem? https://github.com/xiph/opus/issues/221
Transferring media stream to worker via track processor https://github.com/w3c/webcodecs/issues/43