streaming support

[lepinsk]

First off, really neat work here with this! :+1:

I came here via your post on StackOverflow, and I wanted to ask your thoughts on what would be involved in converting this to something that could accept a stream from getUserMedia. You also pointed to kazuki/opus.js-sample, however that's throwing opus_encoder_create failed whenever I try to use it.

I ended up here after playing around with libmp3lame.js for a while, but running into some (potentially unresolvable?) issues around audible artefacts at various bitrates.

Anyway, I guess I was hoping for your thoughts on what would be involved in adding streaming support, and to inquire whether that was something you've taken a crack at. Hope that you don't mind me opening an issue for this.

[Rillke]

Absolutely; I was thinking about a stream encoder, for opusenc.js and flac.js, however I don't know when I'll have the time to tackle that; did you try filing a bug against kazuki/opus.js-sample? Kazuki Oikawa usually replies quickly; in Japanese though.

[Rillke]

In the meanwhile, have a look at chris-rudmin/Recorderjs and recordOpus.

[lepinsk]

Hey @Rillke,

Thanks for the response. I only now noticed the 2nd comment you made – not sure why I never got notified of it.

¯(ツ)/¯

Looks like recordOpus might be what I'm looking for. Fingers crossed.

[lepinsk]

Hey, here's a sort of side question: would you expect, given the same input file, that opusenc.js produces an identical output file to opusenc?

I figured it could be related to differences in component versions, so I cloned a copy of this and set the opus submodule to version 1.1, and opus-tools to 0.1.9, which matches what I've got installed locally, but I haven't yet had luck getting a full build to work.

Anyway, I'm running the same wav file through opusenc.js and opusenc via the command line and it's producing output files that don't hash to the same value. Any thoughts?

[Rillke]

Any thoughts?

In theory the output should be the same, provided the same version of opusenc has been used (and the same version of Ogg and Opus). But what really matters in the end is that the output files pass the opus analyser (opusinfo).

[Rillke]

Hey, here's a sort of side question: would you expect, given the same input file, that opusenc.js produces an identical output file to opusenc?

nope, it's non-deterministic unless you specify and force a --serial number.

[mitar]

I found this ticket. What is current status? I am looking into something which would allow me to record a mic live in the browser, send it to the server, where it would be then in real-time fetched by clients and played back. So a live radio broadcasting.

[amayant]

Yes, me too, but it's very difficult to manage the audio stream. We must create a buffer locally.

[Rillke]

I found this ticket. What is current status?

As you see from the label, this is a new product request. There are several things you need and there might be reasons why there is no solution available, yet:

• You need a stream encoder. So you need to encode small chunks of audio. This usually causes a) a delay b) inferior audio compression than encoding bigger chunks. Opus has some flags controlling this.
• You need to re-sample your audio before passing it to the opus encoder. Those two points are already solved at chris-rudmin/Recorderjs
• JavaScript running in a web worker... if the machine runs out of processor availability just for a moment, data queues on the machine which is recording and you have gaps in speech at those who listen to the live session. You cannot prioritize the web worker thread. And running the encoder in the JS main window thread can be dangerous as it may crash you client's browsers entirely if there aren't enough resources. Just consider an ill-configured desktop with Windows and ngen suddenly starting up.

[mitar]

You need to re-sample your audio before passing it to the opus encoder

Hm, why is resampling necessary? Because opus encoder requires a different input sample rate then what you get from the browser?

JavaScript running in a web worker...

So from your experience is opus fast enough for real-time encoding in JS?

[Rillke]

Hm, why is resampling necessary? Because opus encoder requires a different input sample rate then what you get from the browser?

In theory, you can get a lot of different sample rates from the browser, depending on the browser itself, the operating system and maybe even on which revision of a library is installed on some systems. If the browser dumps on you data in an incompatible rate, you have to resample.

So from your experience is opus fast enough for real-time encoding in JS?

I have never tried this. The opus encoder (opusenc.js) demo runs with approx. 5x speed while having to resample from 41kHz to 48kHz in my browsers. 5x should be sufficient... but as soon as the user opens a different tab, the process or thread running in the old tab gets lower priority. @mitar In case you give it a try, please let me know about the results. That'd be tremendously pleasing to know.

[mitar]

but as soon as the user opens a different tab, the process or thread running in the old tab gets lower priority.

But shouldn't then we use ScriptProcessorNode for encoding instead of a web worker?

In case you give it a try, please let me know about the results. That'd be tremendously pleasing to know.

Currently I am still investigating options. If I do something I will report back.

[mitar]

Hm, but web dictaphone demo records the audio directly into the Opus codec? So we can use built-in codec for this?

[mitar]

Demo: https://mdn.github.io/web-dictaphone/

[mitar]

See also this ticket.

[Rillke]

Hm, but web dictaphone demo records the audio directly into the Opus codec? So we can use built-in codec for this?

Yes, but MediaRecorder API wasn't supported by browsers at the time we made this encoder. And you still have no way to let the user interact with the recording after it has been created.

[Rillke]

There is still no working MediaRecorder in Chrome.

[mitar]

And you still have no way to let the user interact with the recording after it has been created.

What do you mean? You can get a chunks and then do whatever you want with them?

So for my purpose it seems that the best way would be to have MediaRecorder record in Opus and then use WebRTC to send that stream directly to the server, server being the second WebRTC peer (so running node.js WebRTC peer implementation, which I have not yet found any).

It seems this example does that. It seems it works in Chrome?

[Rillke]

What do you mean? You can get a chunks and then do whatever you want with them?

And what do you want to do with opus-encoded chunks? With raw audio data, you could do all kind of processing. With encoded opus blobs, it's a lot harder.

It seems this example does that. It seems it works in Chrome?

It seems so. But it doesn't in Firefox (at least not for me).

[mitar]

Yea, I had issues in Firefox as well. But then Firefox has MediaRecorder. So something seems to be doable here.

[Rillke]

There seems to be a company interested in encoding/ decoding Opus entirely in JS. Supposedly I can publish the stream en/de-coder here after their review. Ping me in case I forget.