search

TNTwise / REAL-Video-Enhancer

Interpolate and Upscale easily on Linux/Windows.
GNU Affero General Public License v3.0
269 stars 19 forks source link

[Feature Request] Image Collage and Precise Frame interpolation, maybe not rendering to lossy formats #4

Closed Melechtna closed 9 months ago

Melechtna commented 10 months ago

rife-v4 allows for specifying the number of frames that should be generated, because often the math for going from one frame-rate to another, especially with image collages, is dead wrong with every interpolator I've ever used, it's almost always short. You could do this automatically of course, the math is quite simple. It's ceil(target_fps * (original_amount_of_frames\original_fps)). We always want to round up to the nearest whole number, or you'll be one short. Conversely, I'm working with image collages for a variety of reasons, and I'm just having a hell of a time getting vapoursynth to do want I want to deal with the jank of transitions. I'd also prefer something other than h264 or 5, as these are always lossy, but could swap something lossless back to an image collage.

TNTwise commented 10 months ago

I plan on adding more codecs, thank you. For the issue, is this what you mean? https://github.com/nihui/rife-ncnn-vulkan/issues/67

Also what codecs would you like to see implemented?

Melechtna commented 10 months ago

No, actually, I did end up there, but, no, I don't think so.

So, if you're using rife-v4 specifically, I'm unsure if v4.1/2/3/etc can still do this, you can do something like this

rife-ncnn-vulkan -x -m rife-v4 -f frame-%010d.png -n 3836 -i input -o output

And it'll interpolate to that EXACT number of frames. The video I'm using as a test at the moment extracts from ffmpeg as 799 frames, and it ran at 12.5fps. So using the equation I posted above, that's 60*(799\12.5)=3835.2, rounding this up, 3836. Once this is interpolated to that many frames, it runs perfectly in sync with the origin video at 60fps, and besides the transition points, is quite smooth

So the formula I've given is for frame perfect conversion from the origin fps to the target fps in all use cases, so no desyncs should ever occur. Conversely however, any time I've used say, the anime one, or just rifev3, which bizarrely seems to bind this to play time, and frankly that makes no sense as a metric to tie frame projection to. With the test footage I'm using, it's consistently about 700 frames below the target, and results in footage that is 16.77% faster than the original.

TNTwise commented 10 months ago

I see, what I do is just take the amount of frames and multiply it by the times of interpolation, and render it out with the frame rate*times. I will look more into it when I am home.

Melechtna commented 10 months ago

I see, what I do is just take the amount of frames and multiply it by the times of interpolation, and render it out with the frame rate*times. I will look more into it when I am home.

I mean sure, but most of them don't accept floating point numbers, which is where that becomes a problem. To do that, you need a simpler calculation 60\12.5, which in this case is 4.8, and it doesn't like that. x5 would obviously be moving to fast, and x4 even worse in the opposite direction.

TNTwise commented 10 months ago

In the latest beta https://github.com/TNTwise/REAL-Video-Enhancer-BETA/releases/latest I have added more codecs and a custom fps increase option. Sadly, lossless encoding does not support transition detection for now, but I am working on it. You are welcome to test and please tell me about any bugs experienced.

TNTwise commented 9 months ago

Added in latest stable, I will close this, but if any issues pop up please open another.