Thank you! Very nice interpolation

jhogsett commented 1 year ago

Hi, Thank You for your great video frame interpolation model!

I wanted to try it out, and compare it to VFIformer. I had previously created a web-based tool: "VFIformer-WebUI". It made sense to me to make a version of my tool that uses your model instead.

I've created "EMA-VFI-WebUI", a version of my application that uses your 2x interpolation model:

https://github.com/jhogsett/EMA-VFI-WebUI

It makes no changes to your code, and just needs to be overlaid on top. The readme file has details on how to install and run it. I think this will make it easy and fun to use your model and also compare it with VFIformer. My original application is at https://github.com/jhogsett/VFIformer-WebUI.

GuozhenZhang1999 commented 1 year ago

Wow! This tool looks fantastic! Thank you so much for recognizing our work, and we'll definitely add a link to your tool in the README file. Personally, I was thinking that it might be even simpler and more efficient if we could compare the effectiveness of different models at the same time. Thanks again for sharing your awesome creation with us!

jhogsett commented 1 year ago

You're welcome! Thank you for your amazing research and model, and thanks for the link!

I have a keen interest in digitized film restoration. I'm very happy to find a model like yours.

I have started working on some comparisons myself, recreating some of my previous work with VFIformer. I'm happy to share some results here when I have them.

Daniel-Kelvich commented 1 year ago

@jhogsett I would be interested to see your comparison!

GuozhenZhang1999 commented 1 year ago

You're welcome! Thank you for your amazing research and model, and thanks for the link!

I have a keen interest in digitized film restoration. I'm very happy to find a model like yours.

I have started working on some comparisons myself, recreating some of my previous work with VFIformer. I'm happy to share some results here when I have them.

Looking forward to seeing your progress! We will also provide a video that compares our method with a few others, and we hope it will be helpful to you!

jhogsett commented 1 year ago

@GuozhenZhang1999 @Daniel-Kelvich

Here's my initial test, which used to evaluate EMA-VFI. Background:

My tool has a GIF-to-MP4 feature
It increases the frame rate using frame interpolation, and increases the frame size using Real-ESRGAN
Note: it uses simple binary interpolation (I plan to use your multi-interference model in a future version)

Original GIF: https://user-images.githubusercontent.com/825994/220933026-3fe6bf25-8be7-490e-a90b-85a151e6b156.gif

VFIformer MP4: https://user-images.githubusercontent.com/825994/224549158-7df76e49-28ec-4a3c-9b2b-4699729bd84f.mp4

EMA-VFI MP4: https://user-images.githubusercontent.com/825994/224548062-4cad649c-5cdb-4f66-936d-e2296eb0fbc8.mp4

(Both examples have been created just now using the latest code. Rate increased 8X, size increased 4X)

What I observe:

EMA-VFI is noticeably smoother, especially around 0:05

GuozhenZhang1999 commented 1 year ago

Thank you for providing such a comprehensive example! Based on the results, it appears that our model indeed has better temporal consistency. If you are willing, please feel free to continue updating your progress on this post. Although we may not reply to each one individually, please know that we greatly appreciate your valuable contribution!

jhogsett commented 1 year ago

You're welcome @GuozhenZhang1999 ! I will post some more examples here for you and others to enjoy. I notice also that the EMA-VFI version looks slightly less dull than the VFIformer version.

jhogsett commented 1 year ago

Here's another similar example.

VFIformer: https://youtube.com/shorts/IZdEITUQx3U?feature=share

EMA-VFI: https://youtube.com/shorts/PLkOUxiFAuk?feature=share

I prefer the EMA-VFI version:

The motion seems smoother and more fluid
The image looks less dull than the VFIformer version
The sudden disappearance of objects seems to be handled better (noticeable near the end)

Original GIF: musicmovesyou-escape-to-music

jhogsett commented 1 year ago

I've just posted a new YouTube video with a live comparison of EMA-VFI, VFIformer (footage inflated to 480 FPS) and the original footage (@ 30 FPS) https://youtube.com/shorts/n-YCv37bU9s?feature=share

jhogsett commented 1 year ago

Here is another video, same format. This time it's a police chase (footage inflated from 24 FPS to 768 FPS). https://youtube.com/shorts/En5VsJNVmnE?feature=share

jhogsett commented 1 year ago

Here is one more: https://youtube.com/shorts/P5TuFfLWIIM?feature=share

This is a carnival at night inflated from 15 FPS to 480 FPS (like inflating a 30 FPS video to 1000 FPS!).

EMA-VFI shows its quality very well, especially near the end, when the fast flashing lights seem to confuse VFIformer.

jhogsett commented 1 year ago

Three videos demonstrating the recovery of original video footage from time lapse videos using VFIformer and EMA-VFI:

Traffic: https://youtube.com/shorts/rwiyvQplQ28?feature=share

Carpenter: https://youtube.com/shorts/Emk3x-8jQNc?feature=share

Airplanes Landing: https://youtube.com/shorts/DRvn-a0u4zE?feature=share

jhogsett commented 1 year ago

The last of my video comparison series between VFIformer and EMA-VFI. For the final videos I chose very challenging content: ocean water, first-person POV, fast movement.

Electric Surfboard: https://www.youtube.com/shorts/BzBHsiEnNxg

Kite Surfer: https://www.youtube.com/shorts/W5otlVo989I

Rollercoaster: https://www.youtube.com/shorts/nb6uTRohPgU

semel1 commented 1 year ago

If you can provide me with your original videos (preferably non-vertical) - I could make comparison using RIFE.

jhogsett commented 1 year ago

@semel1 each of the above YouTube videos includes a link to the royalty-free video used to create the comparison from MotionArray.com. Free watermarked preview versions can be downloaded without needing an account.

In particular I would be interested in seeing a version of this study, where the video has been inflated 32X to 768 FPS: https://www.youtube.com/shorts/En5VsJNVmnE

Edit: here's the link to the video: "The Pursuit Slaloms Left And Right" https://motionarray.com/stock-video/the-pursuit-slaloms-left-and-right-1171307/

semel1 commented 1 year ago

@semel1 each of the above YouTube videos includes a link to the royalty-free video used to create the comparison from MotionArray.com. Free watermarked preview versions can be downloaded without needing an account.

In particular I would be interested in seeing a version of this study, where the video has been inflated 32X to 768 FPS: https://www.youtube.com/shorts/En5VsJNVmnE

Edit: here's the link to the video: "The Pursuit Slaloms Left And Right" https://motionarray.com/stock-video/the-pursuit-slaloms-left-and-right-1171307/

Please take a look at https://youtu.be/mFU8z-DSTds

jhogsett commented 1 year ago

Thank you. I'm seeing a significant loss of detail in your example, especially noticeable in the final part with the close-up of the car grill.

2blackbar commented 1 year ago

Hey man can you do one click nstaler with own venv so we dont have to deal with incompatible dependencies versions clashin?

MCG-NJU / EMA-VFI

Thank you! Very nice interpolation #4