Issues with interpolating <30 fps videos

[cferrier1995]

I'm not observing any issues if the source video is already 60 fps, I tried throwing some very complicated 60 fps videos and they look fine.

The issue seem to occur when the source video is 30 fps or less and has a degree of complicated motion. RIFE tends to fall apart completely and render many weird artifacts. I'm curious if this is just an unsolvable problem at this point in time (low-fps videos with a lot of action just don't have enough data to make informed interpolations) or if there's a series of config options/a better trained model that could fix this.

This is an example of a 30 fps video that RIFE struggled with:, recorded at 4k 30ps on an iPhone: https://drive.google.com/file/d/1SP7c_Rf9gPWNnBwIEM0WG2EhstBCtYW8/view?usp=sharing

I have not found any set of configuration options that make this source video look realistic with RIFE. Another good example would be any first person video of someone skiing, the complicated motion combined with a low framerate proves difficult.

[deama]

This is what I got, looks fine to me? If you're going frame by frame then yeah, you can see some weird stuff, but in motion it looks pretty fine. https://drive.google.com/file/d/19Kk9wFX-hcXDc9WTkMFxNT80XuygKXaa/view?usp=sharing

[cferrier1995]

Interesting. What model and settings did you use? I'll give it another shot.

[deama]

Everything default, though I changed the ssim, scene transition value down to just 0.14. That's probably what the issue was for you? Cause that value is normally 0.5, which is way too high, it will create a type of afterimage of the next frame instead of interpolating it because it thinks it's a scene transition. Look for "if ssim < 0.5:" in inference_video.py

[cferrier1995]

Interesting. This is what it looks like at ssim < .12 if we really push it and do --exp 4 --fps 30

https://drive.google.com/file/d/1GEpR07qKhm3JoSJT5AbTih9D-BVfdx96/view?usp=sharing

If we remove --fps 30 then it looks fine, because the inconsistencies and weirdness are hidden. So I suppose a decent caveat/takeaway here is that it's possible if the source video is 30 fps and there's a lot of motion, it might look strange slow-mo'd.

[deama]

You can try it with that slight modification I did here https://github.com/hzwer/arXiv2020-RIFE/issues/137 maybe it'll improve it.

[cferrier1995]

This did not seem to help

https://drive.google.com/file/d/1o20ikYu6eLbxCzVaU7hLJQdv7i6veMAQ/view?usp=sharing