Not getting a callback for every frame when doing heavy processing

[acgourley]

I'm trying to use this to do some video processing but when I do heavy processing inside the event handler video.on('frame', function(frameIdx, data) {... it doesn't wait for me to finish and thus by the time the main thread can receive an event again many frames have passed.

I could keep the raw frames in storage and then process them later. If this is the only approach, could you suggest a way to do get these frames faster than 1.0x the video playback speed?

[tmarrinan]

Yes. I am doing the demuxing in a separate thread from the main JavaScript thread. Therefore if the video is decoding a frame every 50ms, but your processing takes 200ms, then multiple frames will pass by.

It seems as though you actually need to decode at speeds slower than 1.0x video playback, as speeding up decoding would just make the problem worse.

In demuxworker.cc you see the following:

uint64_t demux_curr = uv_now(uv_default_loop());
uint64_t video_curr = baton->current_frame * baton->frame_time * 1000.0;
int64_t diff = (video_curr - baton->video_start) - (demux_curr - baton->demux_start);
if (diff <= 0) {
    Nan::AsyncQueueWorker(new DemuxWorker(baton, true));
}
else {
    baton->m_Frame(baton->frame_buffer);
    uv_timer_start(&baton->timerReq, uv_DemuxTimer, diff, 0);
}

This basically tells the demuxer to decode one frame immediately if the time passed since last frame is greater than the video playback length for one frame, and to set a timeout for the difference if not enough time has passed.

If you fork the repo, you should be able to create a new mode, DA_NEXT_FRAME, which would decode one frame then go back to DA_NONE. You could then call this function whenever you are done processing the current video frame. Therefore you'd process the frames at the speed of you computation (little computation = faster than playback speed, lots of computation = slower than playback speed).

[acgourley]

Thanks - I'm working on a solution and will PR when done.