Audio playback issues

[Leone25]

Hello,

I'm using jmuxer with node.js to generate a live video feed over which I would like to play audio that is grabbed from an http stream. Here's what I've come up with:

// [...] video is generated, compressed and fed elsewhere
fetch('https://listen.atomic.radio/hitshistory/lowquality', {
    signal: this.abortController.signal,
}).then(res => {
    res.body.on('data', chunk => {
        this.feed({
            audio: chunk,
        });
    });
    return res;
});

The video works fine by it self. But if I enable audio, it lags at the beginning, while some music is playing in small chunks, then it stops playing any music but the video starts working fine.

I've also tried making an audio buffer, then sending all the cached data concatenated at the same time as frame data, but that just made it not play audio at all.

Any help/tips to fix this?

Thx Rico

[Leone25]

Update, I read a bit of the code and noticed that it expects that, when audio data is passed, it should have an aac header at the beginning of the buffer, and it won't try to find it anywhere else.

So my solution was using codec-parser to first parse the stream, and then feed each single frame to jmuxer.

this.audioFetch = await fetch('https://listen.atomic.radio/hitshistory/lowquality', {
    signal: this.abortController.signal,
}).then(res => {
    res.body.on('data', chunk => {
        for (const frame of this.codecParser.parseChunk(chunk)) {
            this.feed({
                audio: frame.data,
            });
        }
    });
    return res;
});

The issue now is that if I use a low quality stream it will be slowed down when playing back (at least on firefox, on vlc it will just play in chunks, while if I use a higher quality stream, it will be slowed down (on firefox, on vlc it will still play in chunks, but smaller silence chunks).

I'm quite confused as to what could be the issue this time... at least on my part I think I did everything right. Any input @samirkumardas?

Thanks gain Rico

[samirkumardas]

@Leone25 First of all, JMuxer is a dumb and very simple muxer, so don't expect too much from it.

It expects each chunk with a proper ADTS header. It does not traverse the byte streams and find the AAC frames from any given position of the stream. You have to do it before you feed it into jMuxer.

[Leone25]

yea, that's what I figured out, and that's why in my second example I'm using codec-parser (npm) to do exactly that, what I don't understand is why it is playing at the wrong speed... Is jmuxer expecting a specific bitrate or something strange?

[Leone25]

ok, I think I figured it out!

The solution is to pass the duration to jmuxer, apparently it's unable to calculate it correctly by it self (in my case since I have both audio and video, I need to do it for both) For audio it was a simple as

jmuxer.feed({
    audio: frame.data,
    duration: frame.duration
});

For video, since I'm using ffmpeg and it's generating an H264 chunk every frame, the duration is the duration of a frame, so

jmuxer.feed({
    audio: data,
    duration: 1000/this.fps
});

Thanks for all your help, and I hope this helps someone else too.

And also thanks Samir for the library!