WebAudio API Sync

[dudewheresmycode]

I'm trying to create a way of syncing to a separate audio file running in Web Audio API. Thinking I could use the timebase of the audio to manually progress the buffer index. Any tips on pulling this off? Note: I'm using the AJAX / HTTP download version, not streaming.

[phoboslab]

That's probably an issue with how scheduleNextFrame is implemented. It should catch up if it detects that it's a bit behind - decoding the next frame at the next possible display refresh. I haven't done any tests to see if it actually does that though.

As you said, to really fix it, you'd have to sync the playback to the Audio's .currentTime somehow. If you implement a frame counter in decodePicture you could calculate the MPEGs currentTime (jsmpeg.currentFrame/jsmpeg.pictureRate) and then maybe just manipulate the .lateTime so decoding catches up.

Do you have a test case for it?

[dudewheresmycode]

Did some tinkering added a avVideoSyncCallback() function that makes sure both audio/video are loaded and starts playback together. Also added a sync function for the audio/video timing:

in scheduleNextFrame()

    this.lateTime = (Date.now() - this.targetTime);

    if(this.audioEnabled){
        var videoTime = this.currentFrame/this.pictureRate;
        var audioTime = jsmpegAudioContext.currentTime-this.audioStartTime;
        this.avSyncTime = (audioTime-videoTime)*1000;
        this.lateTime = this.avSyncTime;
    }

    var wait = Math.max(0, ((1000/this.pictureRate) - this.lateTime) );

    this.targetTime = Date.now() + wait;

Thinking this is probably the route to go.

[shanytc]

@dudewheresmycode I wasn't able to implement your code, I think @phoboslab 's code has changed since, and it spews out a lot of errors... any chance to revisit the code and re-add the code?

[kevinschaul]

I have audio seeming to work in a project using jsmpeg, borrowing @dudewheresmycode's code. Probably shouldn't be merged without some thought, but I'll leave a link to that version here: https://github.com/kevinschaul/jsmpeg/tree/feature/audio

[caruzo]

Hi, where is the actual audio file referenced in the code?

[phoboslab]

Since the new version also support MP2 decoding, audio sync with an <audio> element is not needed anymore.