search

nihui / cain-ncnn-vulkan

CAIN, Channel Attention Is All You Need for Video Frame Interpolation implemented with ncnn library
MIT License
132 stars 15 forks source link

Can it handle large pans? #10

Open noobtoob4lyfe opened 3 years ago

noobtoob4lyfe commented 3 years ago

I'm really enjoying playing with this software. Thanks for making it available!

I'm noticing that it works really well on details and features that move over a small percentage of the screenspace from one frame to the next.

However, I'm also noticing that it seems to freak out when there are pan and tilt motions of the camera that exceed a certain amount of degrees per frame. See here for an example. https://drive.google.com/file/d/1BPvVEIK1u1ngv9607tWEOm-vGNmr-Xug/view?usp=sharing I find this strange because cheap traditional optical flow frame interpolation has no problem with this same shot.
Is this something that can modified in the code to fix? I've tried basically every window size and nothing seems to help. Could it possibly be trained handle these kind of camera moves better? Thanks in advance!

noobtoob4lyfe commented 3 years ago

Anyone have any ideas about what's happening here, or what I might be able to do to improve? Thanks!

routineLife1 commented 3 years ago

Have you tried -t 2048

noobtoob4lyfe commented 3 years ago

Thanks for the reply. Yes, i did try a number of different values there including 2048 and it did not improve things.

routineLife1 commented 3 years ago

I think this should be a problem with the CAIN algorithm

routineLife1 commented 3 years ago

It is recommended to use rife-ncnn-vulkan instead of cain-ncnn-vulkan

noobtoob4lyfe commented 3 years ago

Thanks. I will try that

Hudaldadi commented 3 years ago

well - RIFE is NOT the new CAIN.

CAIN does things different and is much less VRAM hungry.

I hope there will be a working CAIN soon again.

Hudaldadi commented 2 years ago

This looks very promising... https://github.com/google-research/frame-interpolation/