search

hzwer / ECCV2022-RIFE

ECCV2022 - Real-Time Intermediate Flow Estimation for Video Frame Interpolation
MIT License
4.35k stars 432 forks source link

Arbitrary timestep interpolation #330

Open lassefschmidt opened 1 year ago

lassefschmidt commented 1 year ago

Hello there !

First of all, thank you for all the work you have put into this!

I am currently trying out some algorithms for arbitrary timestep frame interpolation and came across your repository, which says in the very first paragraph

It supports arbitrary-timestep interpolation between a pair of images.

Since then, I am trying to get this to work. However, all the recent published models (e.g. the HD_v3 stuff, including what is published under Practical-RIFE) do not support this -- it seems this is only implemented for the RIFE model as published in ArXiV.

But even when I use the ArXiV RIFE model, I cannot load the RIFE_m weights you provide under this link due to shape mismatches. This is a bit unfortunate as a lot of computation power went into the pre-training of those weights. Quote of your paper, pg 8:

We use the Vimeo90K-Septuplet [62] dataset to extend RIFE to support arbitrary-timestep frame interpolation [9,24]. This dataset has 91, 701 sequence with a resolution of 448 × 256, each of which contains 7 consecutive frames. [...] We keep other training setting unchanged and denote the model trained on Vimeo90K-Septuplet as RIFEm.

Is there any chance you can tell me how I can load these weights correctly / provide an updated flownet.pkl file ? Or give me some tips on how to run arbitrary-timestep interpolation with your repo in general?

Thank you!

Best, Lasse

hzwer commented 1 year ago

You can refer to 'benchmark/HD_multi_4X.py' to run that model.

https://github.com/megvii-research/ECCV2022-RIFE#evaluation