search

the-database / mpv-upscale-2x_animejanai

Real-time anime upscaling to 4k in mpv with Real-ESRGAN compact models
Other
325 stars 6 forks source link

Is there a way to simultaneously use AnimeJaNai with RIFE AI frame interpolation? #7

Closed lightc786 closed 1 year ago

lightc786 commented 1 year ago

I'm using rife-cuda in mpv_lazy and wanted to see if I could get AnimeJaNai working alongside it. I'm totally new to using these machine learning models. Is the only way to do it to merge the two desired ONNX models? It's easy to apply anime4k shaders with RIFE so I'm hoping there is a solution to get this working too.

the-database commented 1 year ago

I've done some testing with this. While it is possible to run them simultaneously in theory, AnimeJaNai is so resource intensive that running it together with RIFE is too slow for most hardware to play back in realtime.

One test I tried was upscaling 1080p to 4k with the fastest SuperUltraCompact model and then doubling fps with RIFE, but even an RTX 4090 wasn't fast enough to run it. But it is capable of upscaling 480p to 960p with SuperUltraCompact and then doubling fps with RIFE.

Anyway, even though most hardware can't run it fast enough today, there's no reason not to fully support it. Eventually the hardware should catch up, or there might be software advancements that allow things to run faster. So with the upcoming V2 release, this will be much simpler to set up. There will be a config file for animejanai where you will be able to set up slots, select the upscaling models you want, and choose if you want RIFE interpolation on top. You can see a work in progress of that config file here: https://github.com/the-database/mpv-upscale-2x_animejanai/blob/v2/shaders/animejanai_v2.conf

Aut0Clicker commented 1 year ago

imo 480 to 960 is very adequate. You can then upscale 960 to 4k via Anime4K and you have what I would hope a great image. For higher resolutions I'd probably use a less strong upscaler. Also is it possible to utilize the "normal cores" for REFI and the tensor cores for AnimeJaNai? then the gpu will be utilizing all of its potential and we would get much better frame rate. So we can eseentially use REFI and AnimeJAI with almost no performance penalty?

Also if im not mistaken the new RIFE v2 model works significantly faster, so that should help as well

aloola18 commented 1 year ago

I can do 1080p@24 > x2 with SuperUltraCompact > downscale to 1080p > RIFE to x3. work fine for me and the image did look better than the original. My setup is 3070ti+AMD 5900x

Original: https://cdn.discordapp.com/attachments/290709370600423424/1100859431115313383/image.png

animejanai+RIFE https://cdn.discordapp.com/attachments/290709370600423424/1100859576036884490/image.png