Closed padenot closed 2 years ago
Only the left channel is rendered for now, because that was easier enough to get something going. Not sure if Chrome has all the
copyTo
variants already, and[AllowShared]
, but in any case I have code lying around to interleave/deinterleave if necessary
I was using the wrong google-chrome-unstable
it seems like, I took another one (95) and it doesn't throw anymore.
If I understand the requirement correctly this https://guest271314.github.io/webcodecs/ is how I serialize and deserialize audio with WebCodecs. It is non-trivial resampling the fixed 48000 sample rate that Chromium has hardcoded, especially when the input file is 22050, 1 channel, and I never set Opus encoder sample rate to 48000.
The AudioRenderer sees that the ring buffer becomes empty and crosses its threshold for decoding -- it then proceeds to decode some compressed audio and pushes the PCM to the ring buffer. If the threshold isn't reached, it simply re-schedules itself in the future.
The rescheduling part can be problematic re glitches and gaps between schedules. One option is to utilize WebAssembly.Memory.grow()
to dynamically grow the underlying buffer - within the 4GB Chromium limitation - to write and read to a single contigous block of memory, something like
const initial = (384 * 512 * 3) / 65536; // 3 seconds
const maximum = (384 * 512 * 60 * 60) / 65536; // 1 hour
let started = false;
let readOffset = 0;
let init = false;
const memory = new WebAssembly.Memory({
initial,
maximum,
shared: true,
});
console.log(memory.buffer.byteLength, initial / 65536);
// ...
await readable.pipeTo(
new WritableStream(
{
start() {
console.log('writable start');
},
async write(value, controller) {
console.log(value, value.byteLength, memory.buffer.byteLength);
if (readOffset + value.byteLength > memory.buffer.byteLength) {
console.log('before grow', memory.buffer.byteLength);
memory.grow(3);
console.log('after grow', memory.buffer.byteLength);
}
let uint8_sab = new Uint8Array(memory.buffer);
let i = 0;
if (!init) {
init = true;
i = 44;
}
for (; i < value.buffer.byteLength; i++, readOffset++) {
if (readOffset + 1 >= memory.buffer.byteLength) {
console.log(`memory.buffer.byteLength before grow() for loop: ${memory.buffer.byteLength}.`);
memory.grow(3);
console.log(`memory.buffer.byteLength after grow() for loop: ${memory.buffer.byteLength}`);
uint8_sab = new Uint8Array(memory.buffer);
}
uint8_sab[readOffset] = value[i];
}
if (!started) {
started = true;
aw.port.postMessage({
started: true,
});
}
},
close() {
console.log('writable', readOffset, memory.buffer.byteLength);
aw.port.postMessage({
readOffset,
});
},
}
)
);
What's left to do is to check A/V sync on various OSes, I have only been testing on a Linux desktop with a USB DAC.
This works well for me in Chrome 94 beta with Web Codecs enabled on my Linux box, here's how it goes:
SharedArrayBuffer
-based ring buffer is given to a very simpleAudioWorkletProcessor
(that reads from it) and the control thread (for now the main thread, but we'll be able to move that to a Web Worker)AudioRenderer
pre-buffers some audio and writes the PCM to this ring buffer, on initialization.AudioContext
starts, theAudioWorkletProcessor
starts to read from this ring buffer to output the audio.AudioRenderer
sees that the ring buffer becomes empty and crosses its threshold for decoding -- it then proceeds to decode some compressed audio and pushes the PCM to the ring buffer. If the threshold isn't reached, it simply re-schedules itself in the future.AudioRenderer
is made viasuspend
ing theAudioContext
, so pause/play is supportedGainNode
and an<input type=range>
-- we get smooth volume ramps for free :-).AudioContext.currentTime
, offset by the output latency, converted to microseconds.But it is only a first cut, this is what I'm planning to fix:
copyTo
variants already, and[AllowShared]
, but in any case I have code lying around to interleave/deinterleave if necessaryAudioContext.outputLatency
in Chrome, see https://blog.paul.cx/post/audio-video-synchronization-with-the-web-audio-api/ for details. For now, I picked a number that was kind of OK for my Linux workstation with my sound card, but YMMV. We'll do something fancier, per platform, in a second step, so it looks goodAudioContext
: no need for extreme low latency here.All the code is in PR is from me originally so no license issue. The ring buffer and the server are from https://github.com/padenot/ringbuf.js, but I've just copied/tweaked the necessary files for this demo.