search

baowenbo / DAIN

Depth-Aware Video Frame Interpolation (CVPR 2019)
https://sites.google.com/view/wenbobao/dain
MIT License
8.19k stars 840 forks source link

Tried using it to upscale anime with no results COLAB #113

Open ofeksadlo opened 3 years ago

ofeksadlo commented 3 years ago

What changes should I do in order to covert 30fps anime to 60fps Because I already did everything as default and the video looked the same. When I downloaded the app from here: https://grisk.itch.io/dain-app It had an option for cartoon or real life video but in colab there is no option for cartoon.

And I can't use the app locally becauses it so slow it would take months.

AlphaGit commented 3 years ago

Hi there! This repository is not the one for the DainApp you're using. This is DAIN, the underlying neural network system that Dain-App uses. DAIN is about video interpolation only, it does not support upscaling images.

ofeksadlo commented 3 years ago

Hi there! This repository is not the one for the DainApp you're using. This is DAIN, the underlying neural network system that Dain-App uses. DAIN is about video interpolation only, it does not support upscaling images.

I misspoke all I want is to interpolate frames in an anime. I'm not looking for any fancy upscaling.

AlphaGit commented 3 years ago

Ok! In that case you can use the Colab file, dice you said that you don't want to run it locally.

There is a section with settings near the top. You would likely set TARGET_FPS to 60, which also is the default value.

At some point during the processing of your video it would detect what is the current framerate, it should be 30 so that it matches what you expected.

Give it a try and feel free to ask questions if you have any issue with it!

ofeksadlo commented 3 years ago

Ok! In that case you can use the Colab file, dice you said that you don't want to run it locally.

There is a section with settings near the top. You would likely set TARGET_FPS to 60, which also is the default value.

At some point during the processing of your video it would detect what is the current framerate, it should be 30 so that it matches what you expected.

Give it a try and feel free to ask questions if you have any issue with it!

Well I did gave it a try and the results looks good. Not as good as I expected though. Probabbly because I don't know how to set it for cartoon. And delete duplicate frames.

Currently I'm trying to test it on a real life video to check if the results are any better. Because the default is probabbly designed for real life video rather than cartoon video.

It's a shame the colab version of DAIN doesn't offer as many options as DAIN tool or at least some guidelines to get the same results.

In a perfect environment I would just launch DAIN tool from colab but that's not possible.

AlphaGit commented 3 years ago

Probabbly because I don't know how to set it for cartoon. And delete duplicate frames.

I think the cartoon setting might probably be another pre-trained model. The one we're using is the most generic one, you are right on that one. It's the one that was part of this paper and that gave the best results overall.

Regarding duplicate frames, that's not something that this code is doing. Feel free to modify the Colab to remove them, maybe you can do that with ffmpeg. Since you have a copy of the Colab, any changes you make will not impact the project, so feel free to do that and if you feel like contributing back, also welcome to do so!

These features and others had not been integrated here for two reasons: first, it was never the objective of this repository to provide a final tool. However, I cannot speak in the name of the owner on whether he's interested in pursuing that or not. I helped create the Colab as a quick replacement to DAIN-App when it wasn't so advanced and it was just running DAIN. Since this can be done with DAIN itself, we helped put together the Colab file with a end-to-end proof of concept of how DAIN works. Now DAIN-App has advanced a lot and we have not kept up, nor I know if it's the intention to do so.

In a perfect environment I would just launch DAIN tool from colab but that's not possible.

You can always rent cloud instances optimized for GPU processing. They come at a cost but it's not out of this world. I was able to run some tests with AWS machines for about ~1 USD/hr.

iBobbyTS commented 3 years ago

Check out my repo iBobbyTS/VFIN, The purpose of this project is easy usage of good algorithms. It includes DAIN and Super-SloMo,and BIN is coming soon. Now there's a installation guide in README, I'm planing to add a Colab Notebook to it so that it will be even simpler. It has a lot of functions, for instance: input and output video(with sound)/png/tiff/npy/npz, specifying start/end frame, applying FFmpeg options for video output, pause and resuming...... (You can interpolate 1080p video on P100 with 16G of Vram) I'm still working on VFIN, stay tuned and look forward to new features. Open an issue there if there are any problem using it.

ofeksadlo commented 3 years ago

I think the cartoon setting might probably be another pre-trained model

It makes a lot of sense because I tried it on a regular video (real life) and the results were preety dam good.

Regarding duplicate frames, that's not something that this code is doing. Feel free to modify the Colab to remove them, maybe you can do that with ffmpeg. Since you have a copy of the Colab, any changes you make will not impact the project, so feel free to do that and if you feel like contributing back, also welcome to do so!

I can try but I'm quite satisfied with the results for the real life video. Probabbly using a cartoon model the results will be much better.

You can always rent cloud instances optimized for GPU processing. They come at a cost but it's not out of this world. I was able to run some tests with AWS machines for about ~1 USD/hr.

That's a good idea I'll give it a try

ofeksadlo commented 3 years ago

Check out my repo iBobby/VFIN, The purpose of this project is easy usage of good algorithms. It includes DAIN and Super-SloMo,and BIN is coming soon. Now there's a installation guide in README, I'm planing to add a Colab Notebook to it so that it will be even simpler. It has a lot of functions, for instance: input and output video(with sound)/png/tiff/npy/npz, specifying start/end frame, applying FFmpeg options for video output, pause and resuming...... (You can interpole 1080p video on P100 with 16G of Vram) I'm still working on VFIN, stay tuned and look forward to new features. Open an issue there if there are any problem using it.

I looked into your reposity and it looks amazing. I'm definitely will try utilizing it.

iBobbyTS commented 3 years ago

By the way, you can use the models provided in Dain-app on DAIN too. in VFIN, use the following argument to do that. -md path_to_the_model.pth

ofeksadlo commented 3 years ago

By the way, you can use the models provided in Dain-app on DAIN too. in VFIN, use the following argument to do that. -md path_to_the_model.pth

Yeah but how I know which one of the model I need for cartoon?

iBobbyTS commented 3 years ago

Download DAIN-app, extract the archive, there's a folder called "model_weights", the models trained by the developer of DAIN-app are in there, upload them to Google drive, now you can load the model from Google Drive by -md. For the latest version 0.48, there's one extra model called "experimental.pth", check if that works better than "best.pth" I never used DAIN-app because don't have a local Nvidia GPU.

ofeksadlo commented 3 years ago

Download DAIN-app, extract the archive, there's a folder called "model_weights", the models trained by the developer of DAIN-app are in there, upload them to Google drive, now you can load the model from Google Drive by -md. For the latest version 0.48, there's one extra model called "experimental.pth", check if that works better than "best.pth" I never used DAIN-app because don't have a local Nvidia GPU.

Ty I'll give it a try btw I doubt most home computers can handle DAIN app you'll need more than just NVIDIA GPU you need a strong NVIDIA GPU.

ofeksadlo commented 3 years ago

Download DAIN-app, extract the archive, there's a folder called "model_weights", the models trained by the developer of DAIN-app are in there, upload them to Google drive, now you can load the model from Google Drive by -md. For the latest version 0.48, there's one extra model called "experimental.pth", check if that works better than "best.pth" I never used DAIN-app because don't have a local Nvidia GPU.

The highest version of DAIN-APP I managed to find was 0.40 But I hardly think it's matter because there is an option for cartoon in the DAIN-APP Even when there is only one model in "model_weights" folder. So I think the different is how they inference the model rather than using an entire different one.

iBobbyTS commented 3 years ago

In fact, the latest version is 0.48, here's their official page for updates: DAINAPP There are two models in this one, but as you said, it can interpolate cartoon with only one model, so this might be what I'm going to work on next.