search

bloc97 / Anime4K

A High-Quality Real Time Upscaler for Anime Video
https://bloc97.github.io/Anime4K/
MIT License
18.54k stars 1.35k forks source link

Live action version? #153

Open satvikpendem opened 2 years ago

satvikpendem commented 2 years ago

Is there something like Anime4k but for live action videos like movies and TV shows? It works great on anime and other animated media but doesn't work so well for live action of course, so I'm wondering what other projects like Anime4k I should look for.

bloc97 commented 2 years ago

There's a few of them, like FSRCNNX and ravu, but they are based on older machine learning architectures/training methods that do not take in account video compression and other degradations. However they do work perfectly well on uncompressed (or very high bitrate) video.

I don't know how effective it would be to train a restore shader for live action but it might be a future project.

satvikpendem commented 2 years ago

I'm actually interested in perhaps doing this myself, any pointers I could look at for how you did it with anime? I assume if we get enough training examples, we could substitute the anime training data with live action data and see how it fares. That's just my assumption however, could be wrong.

christoph-heinrich commented 2 years ago

You might also want to have a look at AviSynth AiUpscale. I've just added it to my config and from my limited testing so far I get almost identical results as with FSRCNNX + adaptive sharpen (curve_height 0.5) with identical GPU usage afaict.

Regarding training data you should start off with very high quality source material and then generate training data from it in a variety of bitrates (and maybe even different codecs) at whatever scaling factors you want to support. That way it should be able to deal with compression artifacts.

satvikpendem commented 2 years ago

@christoph-heinrich take a look at Nvidia Image Scaling, I've been using it in mpv (https://gist.github.com/agyild/7e8951915b2bf24526a9343d951db214) and it's a lot sharper than AviSynth AiUpscale. For being a non-machine-learning shader, it's actually very impressive.

I'll collect training data as you mentioned, that's a good idea to segment into various codecs. Is there a way to have a drop in replacement with Anime4k, ie is there a folder of training data that Anime4k was trained on where I can simply replace the previous training data with my live action one? Or is that not possible?

christoph-heinrich commented 2 years ago

@satvikpendem Thanks for the tip. NVScale introduces some haloing, but AviSynth AiUpscale (medium sharp) makes compression artifacts more prominent, so it depends on the video which one ends up looking better. AiUpscale usually gets closer to what a better quality source would look like imo.

segment into various codecs

Presumably the same model should be able to handle different codecs as long as it gets trained with a variety of them, but as usual with neural networks, it's probably best to find what works through trial & error.

is there a folder of training data that Anime4k was trained on where I can simply replace the previous training data

I've never tried training Anime4k myself or looked at the source code, but since it has to take the training data from somewhere, you can probably just look at the code and replace the path. Due to the increased complexity over lineart I'd assume you'll have to use a bigger network then Anime4k to get decent results. I also don't know how well the multi stage approach from Anime4k lends itself to non-animated content.