Can't read opus encoded ogg file

[larsbeck]

Hi,

I am having trouble reading in the attached file, which works fine in, say VLC or Chrome. Here is two ways that I tried to decode it and the behavior that I get:

        var decoder = OpusDecoder.Create(48000, 1);
        using (var file = File.OpenRead("example.ogg"))
        {
            OpusOggReadStream oggIn = new OpusOggReadStream(decoder, file);
            while (oggIn.HasNextPacket)
            {
                short[] packet = oggIn.DecodeNextPacket();
                if (packet != null)
                {
                    byte[] binary = ShortsToBytes(packet);

                }
            }
        }

This will skip the while loop because HasNextPacket always returns false.

        var decoder = OpusDecoder.Create(48000, 1);
        using (var file = File.OpenRead("example.ogg"))
        {
            var buffer = new byte[1024];
            int bytesRead = -1;
            while (bytesRead != 0)
            {
                bytesRead = file.Read(buffer, 0, buffer.Length);
                buffer = buffer.Take(bytesRead).ToArray();
                int numSamples = OpusPacketInfo.GetNumSamples(buffer, 0, buffer.Length, 48000);
                short[] pcm = new short[numSamples * decoder.NumChannels];
                int samplesDecoded = decoder.Decode(buffer, 0, buffer.Length, pcm, 0, numSamples, true);
                byte[] bytesOut = ShortsToBytes(pcm);
            }

        }

This will decode a few thousand bytes and then show a "corrupted stream" exception. However, even the decoded PCM frames aren't decoded correctly. When I play them back with NAudio for example it is just noise.

Am I doing something fundamentally wrong that I am not seeing?

Thanks in advance!

I had to zip the file, because Github doesn't allow for ogg to be attached.

example.zip

[lostromb]

The problem is that this isn't an Ogg container file, it's a WebM. From FFprobe:

Input #0, matroska,webm, from 'example.ogg':
  Metadata:
    encoder         : Chrome
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0(eng): Audio: opus, 48000 Hz, mono, fltp (default)

I'm assuming you're using Chrome itself to encode this file. I'm afraid I don't know anything about how that works, or whether you can coerce it to generate .ogg instead. If you are just experimenting, FFmpeg is a great way to generate properly formatted files. In this case, you would do:

ffmpeg -i "example.webm" -c:a copy -map 0:a:0 "out.opus"

But that probably doesn't help you for real scenarios because if you have ffmpeg you could just decode opus with that...

[larsbeck]

Yes, you are totally right. I generated this through Chrome and was hoping to decode the stream with your library. I'll dig into how those containers work then. Thanks for investigating it!

[kylewhittington]

@larsbeck Did you ever figure out how to read the Chrome generated webm opus files? I'm trying to get them into PCM via C# and I'm hitting the same issue you were having. It needs to be happen on the fly - which I'm achieving fine with OGG encoded files from Firefox.

[lostromb]

You would need a WebM demuxer. A cursory search doesn't show me a pre-existing one written in C#, just a bunch of wrappers for ffmpeg, so you may be out of luck for now

e: nm, I see what you mean now. You're wondering if Lars ever implemented such a demuxer. I'll admit that having a Concentus.MatroskaFile library would be pretty handy if someone had written the code already

[larsbeck]

@kylewhittington I believe the last thing I looked into was https://github.com/Ruslan-B/FFmpeg.AutoGen in order to directly decode a webm stream in memory, but I didn't put much time into it. Instead I used a wav stream. That sort of worked, but in the end it was too much bandwidth to be useful for our use case.

[kylewhittington]

@larsbeck Oh man... yeah, I'm in a decoding / browser compatibility nightmare. Using getUserMedia to generate audio data to pass to Google Speech to Text. Google's support of Opus is pretty much non-existent (even though they say it is) so we're needing to send it along as PCM (sending them opus just doesn't work). When I found this library it was a godsend and I got it working sweet with Firefox, but then Chrome is unhappy. I think what I'm discovering is that the getUserMedia cross browser support is patchy at best when wanting a consistent encoding... and it appears not many people are streaming client-side web captured audio into an external service (which is expecting a specific encoding).

I'll keep digging :-)

[larsbeck]

@kylewhittington Could you use PCM frames directly from the browser like in this post? http://papers.traustikristjansson.info/?p=486

Here is a snippet that I used back in the day (Typescript): `
audioContext: AudioContext; public Initialize() { let websocket = new WebSocket("ws://YOUR_IP:YOUR_PORT/YOUR_ENDPOINT"); websocket.binaryType = "arraybuffer"; let session: MediaStreamConstraints = { audio: true, video: false }; this.audioContext = new AudioContext(); //local to remote navigator.getUserMedia(session, stream => { let audioInput = this.audioContext.createMediaStreamSource(stream); let bufferSize = 0; // let implementation decide let recorder: ScriptProcessorNode = this.audioContext.createScriptProcessor(bufferSize, 1, 1); recorder.onaudioprocess = e => { if (websocket.readyState == websocket.OPEN) { let left = e.inputBuffer.getChannelData(0); let arrayBuffer = this.convertFloat32ToInt16(left); websocket.send(arrayBuffer); } }; audioInput.connect(recorder); recorder.connect(this.audioContext.destination); }, error => { }); //remote to local websocket.onmessage = ev => { let arrayBuffer: ArrayBuffer = ev.data; if (arrayBuffer.byteLength <= 0) return; this.audioContext.decodeAudioData(arrayBuffer, data => { let bufferSource = this.audioContext.createBufferSource(); bufferSource.buffer = data; bufferSource.connect(this.audioContext.destination); bufferSource.start(); }); }; }

convertFloat32ToInt16(data: Float32Array) {
    var l = data.length;
    var buf = new Int16Array(l);
    while (l--) {
        buf[l] = Math.min(1, data[l]) * 0x7FFF;
    }
    return buf.buffer;
}
convertInt16ToFloat32(data: Int16Array) {
    var result = new Float32Array(data.length);
    data.forEach((sample, i) => result[i] = sample / 0x7FFF);
    return result;
}

`

[kylewhittington]

@larsbeck amazing - I'm definitely going to try that - I wasn't aware of being able to get the PCM frames like that. The problem I've been finding with pushing PCM from client to server (using SignalR - basically just a .NET wrapper of WebSockets) is packet size. I don't think I have many other options so I think it will just be a case of getting the buffer small enough to satisfy those requirements and hope for the best.

Thank you so so much for the advice and help with this... I've been working on this for over a week now and it's been incredibly frustrating, given I thought it would only be a day or so of effort to get cross browser compatibility!

[larsbeck]

@kylewhittington Yeah, I hear you. For a .NET developer this is a world of pain. TypeScript made it slightly better, but you still have the issue of multiple 'runtimes'/implementations of APIs. And yes, David Fowler et al. said that SignalR should not be used for streaming data if I remember correctly. On the other hand, if you know that it uses WebSockets underneath (and does not fall back to one of the other techniques supported by SignalR, such as long polling) then it is worth a try as you are saying. Good luck!

[tomh333]

@larsbeck I know this is a while ago, but after messing around for a week in browser audio hell (as you say a world of pain for a .NET developer) and a lot of wasted time, you're comment (PCM frames directly from browser) finally made it all work! So thanks a lot!