MFlowFps and MBlockFps

[ghedo]

Are there any plans on porting the MFlowFps and MBlockFps functions?

[dubhater]

If you plan to use them, I can probably add them.

[ghedo]

Yeah, I'd use them. I was porting a few avisynth scripts and noticed that they are missing. Unless you can suggest a better alternative it'd be nice to have them.

[dubhater]

Okay.

[ghedo]

Any news on this?

[dubhater]

No news at this time.

[bjorn]

Aw, I was really looking forward to be able to increase the framerate on some videos after getting VapourSynth, vapoursynth-editor, ffms2 and finally vapoursynth-mvtools to work on Linux. But then I got stuck on a missing FlowFps. I'm not sure if there are other ways to do the same, but if not I'd also really look forward to a port of that one.

How complicated would this be? Is it something I could help with? Maybe I could start by looking at the differences between other ported files and their original verions.

Edit: Hrm, those files look very different.

[bjorn]

I posted a small bounty at https://www.bountysource.com/issues/5251616-mflowfps-and-mblockfps

[dubhater]

Cool. Sure, I can do it now rather than some other time, if you insist. :p But I have no way to cash that. Maybe you can buy an ebook instead. http://pastebin.com/Em3JuG5J

[bjorn]

Hmm, how come you can't cash it? They cash out to PayPal, Bitcoin, Ripple, Mastercoin or Check...

I could buy you an ebook as well, but you'd have to tell me how to do that.

[dubhater]

It takes a week to port a filter when you only touch it five minutes a day...

[bjorn]

It's alright @dubhater, with 5 minutes a day I am surprised you got it done at all! I look forward to trying it out soon.

[ghedo]

Cool, thanks!

I just tried it with the following script with the mpv vapoursynth filter:

import vapoursynth as vs
core = vs.get_core()

clip  = video_in
super = core.mv.Super(clip, pel=2)
bv    = core.mv.Analyse(super, isb=True)
fv    = core.mv.Analyse(super, isb=False)
clip  = core.mv.FlowFPS(clip, super, bv, fv, 25, 1)

clip.set_output()

But all I get is a "floating point exception". Here's the backtrace:

Program received signal SIGFPE, Arithmetic exception.
0x00007fffc661e886 in mvflowfpsCreate (in=in@entry=0x156df50, 
    out=out@entry=0x11f97a0, userData=<optimized out>, core=0x1566600, 
    vsapi=0x7ffff33ec800 <vsapi>) at src/MVFlowFPS.cpp:804
804     d.fa /= fgcd;
(gdb) bt
#0  0x00007fffc661e886 in mvflowfpsCreate (in=in@entry=0x156df50, 
    out=out@entry=0x11f97a0, userData=<optimized out>, core=0x1566600, 
    vsapi=0x7ffff33ec800 <vsapi>) at src/MVFlowFPS.cpp:804
#1  0x00007ffff31d1f71 in VSPlugin::invoke (this=this@entry=0x1575180, 
    funcName="FlowFPS", args=...) at ../src/core/vscore.cpp:1252
#2  0x00007ffff31c17dd in invoke (plugin=0x1575180, name=<optimized out>, 
    args=0x156df50) at ../src/core/vsapi.cpp:459

(I left out all the python and mpv lines).

It appears that the getVideoInfo()method returns both fpsNum and fpsDen as 0:

(gdb) p *vsapi->getVideoInfo(d.node)
$15 = {format = 0x124c2f0, fpsNum = 0, fpsDen = 0, width = 1920, height = 800, 
  numFrames = 0, flags = 0}

which means that later numeratorOld and denominatorOld are 0 as well.

Is this a bug in mvtools/vapoursynth/mpv or am I doing something wrong?

[dubhater]

Well, there is a bug in FlowFPS. I should consider the case where fpsNum and fpsDen are zero and avoid that division by zero.

However, this filter needs the input clip to have a frame rate in order to function. mpv doesn't really have a notion of frame rate, so it sets those variables to zero (which is perfectly legal). Make that

clip = core.std.AssumeFPS(video_in, x, y)

and it should work.

[ghedo]

Yep, that works great, thanks!

[bjorn]

Worked fine for me, so if you have trouble claiming the bounty feel free to e-mail me at bjorn@lindeijer.nl about it.

Btw, I was very interested in using this as an mpv filter as @ghedo mentioned to avoid having to preprocess movies when watching them. However, I could not get it to work for smooth framerate doubling. Is that something that can't work in this mode?

[ghedo]

The problem I think is that you'd need a supercomputer to run this in real-time. I use mpv's -o option to convert the video before watching it.

I think MVBlockFps is supposed to be faster than MVFlowFps (it can also create artifacts though) so if it gets implemented that may help.

[bjorn]

@ghedo Well, it depends on the resolution of course, but I could re-encode the video with the filter applied at over 120 fps. When I use vspipe with mpv - the CPU usage is "only" about 200% (out of 800%). So it shouldn't be a CPU issue, and I rather think some parameter is wrong or information is missing from the input video to apply the smoothing correctly.

The latter option doesn't work so well because then mpv can't seek anymore. Also, I still need to figure out how not to strip the audio.

[ghedo]

For reference, here is the script I use to convert to 60 fps (you may need to adjust the src_fps to whatever your source video fps is):

import vapoursynth as vs
core = vs.get_core()

src_fps = 24
dst_fps = 60

clip  = core.std.AssumeFPS(video_in, fpsnum=src_fps)
super = core.mv.Super(clip, pel=2)
bv    = core.mv.Analyse(super, isb=True, overlap=4)
fv    = core.mv.Analyse(super, isb=False, overlap=4)
clip  = core.mv.FlowFPS(clip, super, bv, fv, dst_fps)
clip  = core.std.AssumeFPS(clip, fpsnum=dst_fps)

clip.set_output()

Which works with mpv -o but it takes quite some time. Maybe playing with the functions arguments may help.

[dubhater]

overlap=4 makes it slow. Also note that FlowFPS is not multi-threaded.

To make it work correctly in mpv, FlowFPS needs to set each frame's duration according to the output frame rate. I'll do this soon.

[dubhater]

To check if incorrect frame durations are the problem, call AssumeFPS after FlowFPS, like it the script @ghedo posted above.

[bjorn]

Well, I tried the script by @ghedo now but while it does improve the framerate, the interpolated frames are basically just a plain 50% blending between two frames rather than a correctly derived image based on the pixel movements. Only when I explicitly specify the input video using core.ffms2.Source does the VS script produce the right output (but then it doesn't work as mpv filter, of course).

So it seems to me that there is some difference between the video as ffms2 source or as mpv video_in that prevents FlowFPS from working correctly in the latter case.

[dubhater]

It's a bug in Analyse. It does not (yet) handle correctly clips with unknown length.

[dubhater]

It should be fixed now. The script posted by @ghedo appears to work. (The second AssumeFPS is no longer needed, btw.)

[bjorn]

It's working quite well, thanks! This way I can avoid the lengthy conversion process and avoid at least part of my hard-drive space problems. For HD movies it is unfortunately not fast enough even when dropping the "overlap=4".

[John-Gee]

Hello,

I've actually been waiting for a while for this, so I'm pretty excited about it! Since I saw that plugin for windows I wanted to try, but i don't have windows so...

Alas, I tried with mpv and a 640x480 xvid, and it was way too slow with a lot of framedrops (I have an i5 750 so not the best CPU but not super old either). I used the 2nd script provided by ghedo, without the last AssumeFPS. I've tried with and without overlap=4, but it was the same. I'm currently outputting to a file mpv's pre-processing result to see how it looks, but it's not really the goal.

well... it worked, but the video seemed a lot more blocky than the original, I'll try with overlap=4 this time.

What can I do to make it fast enough to play real-time non hd?

Thanks for this!

[dubhater]

Increasing the block size in Analyse might help. The default is 8x8.

[John-Gee]

Actually I didn't pay attention enough, it's only the saved output of mpv (with -o) that looks like that, in real time it looks fine so I assume it is related to mpv. I'll still look for a way to get it running fast enough for real time. Thank you!