Interpolate a video with variable frame drops?

[tmladek]

I have a video that was created on a machine which on several occasions started dropping frames due to overload.

This means that there's some portion of good video, which needs no processing (and I can cut it out using ffmpeg just fine) but then there's portions with what can best be described as "variable framerate" - frames changing semi-randomly, at an apparent rate of anything from 1 FPS to 30 FPS.

Is there any way to process it and "fill in the gaps" with butterflow? The frames themselves are seemingly accurately timed, simply with a variable amount of identical frames in between them.

I checked out the Example Usage, and it hints at using ffmpeg's "decimate" filter - however, it seems that decimate's expected use-case is removing duplicate frames at regular intervals, so it likely wouldn't be of much use here. Also, I am not sure that butterflow's algorithm can even handle a variable framerate, maybe it assumes a constant FPS and works off of that.

My tests so far have only ended up with what seems to be a single new interpolated frame:

[butterflow:INFO]: Final output frames: 3599 source, +1 interpolated, +0 duped, -0 dropped

Same with the -sm option - so it seems that with the video, as is, isn't a very good input for butterflow.

Any pointers, clues? Anybody with a similar issue? Thanks in advance!

[dthpham]

You're right. BF performs all pre-calculations and rendering based on the assumption that all inputted videos are of a constant frame rate. There's no logic in the program to handle variable frame rates.

[tmladek]

Aw shucks. Thanks for the message though!

How difficult do you think it would be to add? Not requesting a feature, more asking about the feasibility, how critical is this assumption for butterflow, if I were to dive into the code...

[dandelany]

@tmladek If you know the exact times/lengths of the gaps to be filled, I've had some mixed success using the "regions" feature to fill in variable gaps: https://github.com/dthpham/butterflow/blob/master/docs/Example-Usage.md#multiple-regions

[dthpham]

Yep you’re going to have to step through your video frame by frame and use the regions feature to “fill in gaps”/make BF interpolate frames in specific areas in the video. But BF wouldn't know if the two frames it's interpolating between are identical so you're going to have to edit them out some way. Otherwise you're just going to get a lot of duplicate frames.

[dthpham]

Handling variable frame rates would require a rewrite I think. One would have to interface with the FFMPEG API directly, to inspect time stamps, instead of assuming constant frame rate, and then interpolate and write frames based off of that info.

[wagesj45]

@tmladek If you're still looking for a solution, I think @dthpham is on the right track, though actually performing this might be more trouble than the footage is worth. If it is priceless family footage that cannot be replaced, maybe it will be worth it. If you could just recapture the footage, then maybe that would be the way to go. If the framerate is truly that mixed, you might want go look at using ffmpeg to split each frame individually and try to detect scene changes using the select method with a very very low threshold. You could also load the file into a VSE such as Blender. You can probably use others, but I'm familiar with this program and its VSE offers frame-perfect editing. You could then go through frame by frame and use that to filter out the duplicate frames and find the exact timings to put into butterflow regions. Or if accuracy isn't super important and you're using Windows, you can use Butterflow UI and its built in region editor to run butterflow against the footage. (Full disclosure I wrote that.)