search

n00mkrad / flowframes

Flowframes Windows GUI for video interpolation using DAIN (NCNN) or RIFE (CUDA/NCNN)
GNU General Public License v3.0
1.39k stars 115 forks source link

Request: Integrate IFRNet #204

Open dillfrescott opened 1 year ago

dillfrescott commented 1 year ago

IFRNet: https://github.com/ltkong218/ifrnet

Even has an ncnn version: https://github.com/nihui/ifrnet-ncnn-vulkan

It seems to score much higher than even XVFI according to the papers at: https://paperswithcode.com/sota/video-frame-interpolation-on-vimeo90k (scoring second against many algorithms)

This would be a wonderful addition to flowframes!

n00mkrad commented 1 year ago

Already did an experimental implementation, it's mediocre.

Will still make it available soon

dillfrescott commented 1 year ago

Ah, okay!

dillfrescott commented 1 year ago

What do you think is the best interpolation method right now then?

n00mkrad commented 1 year ago

RIFE still

FelixKainz commented 1 year ago

Second this. IFRNet seems to be promising. @n00mkrad is RIFE really still the best interp method? Or do you rather mean the best real-time method?

n00mkrad commented 1 year ago

Second this. IFRNet seems to be promising. @n00mkrad is RIFE really still the best interp method? Or do you rather mean the best real-time method?

RIFE still

dillfrescott commented 1 year ago

@FelixKainz After more testing, IFRNet on tta mode (the -x option on the cli) seems to greatly outperform rife even with rife's spatial and temporal tta modes on.

Im very impressed with IFRNet so far. I am using the IFRNet_L_Vimeo90K model. Because it contains a larger dataset than the GoPro model.

A downside being it takes a lot longer...

FelixKainz commented 1 year ago

IFRNet on tta mode (the -x option on the cli) seems to greatly outperform rife even with rife's spatial and temporal tta modes on.

Intersting. How much longer does it take? Did you try to find the point of deminishing returns as in how long will each take to produce a similar result? If it takes IFRNet only like two or three times as long it would still be a viable alternative for non-real-time use cases.

If you can give more details on your testing, I would appreciate to hear @n00mkrad's take on it!

dillfrescott commented 1 year ago

I have only tested it on 2d animation. So it may be slightly biased. It seems to improve coherency quite a bit versus non tta mode. But the speed reduction is quite insane. I'd say it's probably not worth it unless your footage is a minute or less, or unless you had multiple powerful cards.

hithereai commented 1 year ago

IFRnet ducks. At least according to my tests which were sd generated animations mostly.