search

libretro / common-shaders

Collection of commonly used Cg shaders. These shaders are usable by either HLSL and/or Cg runtime compilers. The cg2glsl script will translate most of these into GLSL shaders.
http://www.libretro.com
1.03k stars 250 forks source link

Waifu2x ported as a shader #75

Open Mickevincent opened 8 years ago

Mickevincent commented 8 years ago

I was told by a forum member at ngemu to try and ask here. I wonder if anyone can port the Waifu2x upscaler and denoiser as a shader to use for PS1 emulation. You can see the result of Waifu in action here, and it looks amazing. FFIX- http://i.imgur.com/zR1kjlE.jpg

FFVII http://i.imgur.com/CzRczDb.jpg

FFVII - 2 http://i.imgur.com/X27lVJ5.jpg

FFVII - 3 http://i.imgur.com/fYwxIBY.png

Seriously this needs to be done. If it is possible to play pre-rendered PS1 games like this (and well, realtime would also benefit) it just must be done. In worst case I could pay a little if the hussle is too much. Here is a URL to a Google translated Waifu2x site

https://translate.googleusercontent.com/translate_c?depth=1&hl=en&prev=search&rurl=translate.google.se&sl=ja&u=https://github.com/lltcggie/waifu2x-caffe/releases&usg=ALkJrhivu32FwUwuPxodRAFRTS4bRFPimA

/Mike

inactive123 commented 8 years ago

@Hyllian made something called NEDI which tries to look like this.

Anyway, a real port of Waifu2x is impossible with the current shader spec. It is a neural network algorithm-driven filter. You'd need OpenCL, CUDA, compute GL/Vulkan to replicate something like that.

leilei- commented 8 years ago

There is a GLSL implementation, but it's still too slow for real-time generation for games. It's more practical when used as a massive batch operation for an external texture pack (which Mednafen-PSX obviously doesn't support and probably won't support)

Also a lot of the PSX texture data are in chunks and sometimes really too posterized/aliased for waifu2x to be effective on (it only really shines with antialiased broad strokes, such as........ animes)

inactive123 commented 8 years ago

Texture replacement in Mednafen/Beetle PSX might be a fair proposition when we are further ahead with the GL renderer. Right now the main priority with the GL renderer is fixing the bugs and getting rid of the bottlenecks.

inactive123 commented 8 years ago

Hmm, looks like that repo only has a MSVC solution. I will try if I can get it to compile and run on Linux/OSX.

Mickevincent commented 8 years ago

Thanx for your answers. I guess I'll have to wait longer for something like Waifu on PS1 games.

leilei- commented 8 years ago

Waifu2x in even PC games with the proper backend support isn't practical either. I forgot to mention it's also bad with tiled textures.

Mickevincent commented 8 years ago

Maybe it works better on pictures from games than it would in the game itself real time? I don't know... I just... need to have something that can make my old pre-rendered rpg's to look like the pictures I have shown. I have tried with ReShade but I can't get it to look like that. Maybe I would need a denoise filter with it I don't know..

hizzlekizzle commented 8 years ago

I think Hyllian's shaders are going to be your best bet for this in real time. Check out the fast-bilateral-super-xbr presets: https://github.com/libretro/common-shaders/tree/master/denoisers (here's a pic) or the aforementioned NEDI (here's a pic). If you're using increased internal resolution, you could also check out the super-xbr-3d presets, which only apply the effect to the prerendered backgrounds/HUD and not the uprezzed polygons.

Mickevincent commented 8 years ago

Thank you. But I don't know. From what I have seen so far it very far away from what I want. I could just try it but Retroarch gave me a headache last time so I deleted everything. I mean I don't know if it's worth it. Will it give me anything close to this quality with those shaders? http://i.imgur.com/fYwxIBY.png Because so far the most I have seen is kinda messy, sadly :(

hizzlekizzle commented 8 years ago

I took a screenshot from FF7 and ran it through waifu2x, which runs slower than real-time, and then did the same thing with a shader preset I just made (nedi/fast-bilateral-nedi.cgp, also at 2x scale to match waifu; runs full speed on Intel HD3000 gfx) and here's the result, unlabeled so you can give them the pepsi challenge: http://imgur.com/a/ixTqZ/all

Mickevincent commented 8 years ago

The right one is Waifu because it has less artifacts. But sure, just there the difference is not that big, at least not on that certain background. But try some other places and you will see. I mean take this as an example

http://i.imgur.com/X27lVJ5.jpg

It will absolutely not look like that, at least not with the filters I have tried. Maybe you can try it with the filter you made?

Hyllian commented 8 years ago

Even that glsl implementation isn't pure shader. The model handler is in C# or C++ and the model is passed to the shader by uniforms.

Not possible with current Retroarch shader specs.

On Sun, May 29, 2016 at 4:15 AM, Mickevincent notifications@github.com wrote:

The right one is Waifu because it has less artifacts. But sure, just there the difference is not that big. But try some other places and you will see. I mean take this as an example

http://i.imgur.com/X27lVJ5.jpg

It will absolutely not look like that, at least not with the filters I have tried. Maybe you can try it with the filter you made?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/libretro/common-shaders/issues/75#issuecomment-222346541, or mute the thread https://github.com/notifications/unsubscribe/AAyZbG0tIgLNg_4M9aa71dtsyMqPy5UIks5qGTz7gaJpZM4IpHsI .

Mickevincent commented 8 years ago

How about if one made a complete new gpu plugin for ps1 emulation and then implemented some kind of waifu filter?

hizzlekizzle commented 8 years ago

That's a pretty tall order, and it still wouldn't run anywhere near real-time.

Anyway, here are a couple more comparisons between waifu and fast-bilateral-nedi: http://imgur.com/a/GX3qU/all http://imgur.com/a/TZT9Q/all There are definitely differences--every upscaler has a signature, and waifu's is quite pleasant--but the differences are pretty small, particularly considering the speed difference. It's also important to recognize that the "artifacts" in the fast-bilateral-nedi shots are actually preserved details that waifu blasts out in favor of flat gradients and sweeping lines. Waifu is just not designed for maintaining detail in this sort of application, though its straight, smooth outlines and high contrast look nice.

Mickevincent commented 8 years ago

That is actualy rather nice. Is it possible for you to upload the exact shaders for me? Is it retroarch? I dont know if I can get it to work but I'll try.

Mickevincent commented 8 years ago

So I tried to upscale the same picture and it do looks better. I do it with these settings

Here it is with some others

http://imgur.com/a/zzdBm

I have never seen these backgrounds before in such high quality. Such an amazing filter.

hizzlekizzle commented 8 years ago

It is indeed RetroArch, and you can download the shader preset through RetroArch's online updater (once you update the Cg shaders, it's in the 'nedi' subdir).

Those shots look nice but you're losing a ton of detail to posterization, and it looks like there's a lot of color post-processing going on. You can get reasonably close to that in RA by adding in the 'natural vision' shader and cranking up the bilateral blur: http://imgur.com/a/hJo6E/all But anyway, the long and short is: waifu2x can't run real-time. If you want to get pretty close, this is how you do it. If not, you're out of luck.

Mickevincent commented 8 years ago

I have talked with the developer of waifu. He is gonna release a 2x faster version. Look its not that I don't believe you but I can't let go of the Waifu look. Someone said pherhaps, maybe, with alot of optimisation, changes beeing done and a strong gpu, it could (maybe) run. If you are up for it, even tho you bassicly said that it cant be done, I'am opening a Kickstarter with a reward for whoever does this. Of course it might not even be approved. Altough the Glide64 driver was approved. This is probably my last try. So just say no in worst case :)

leilei- commented 8 years ago

To clarify, he did say it still wouldn't be good for real-time. Real-time neural network scaling is way, WAY easier said than done and it's not as trivial as you think it is. A Kickstarter won't help.

ghost commented 8 years ago

Still WAY too slow, why the heck are you pursuing this?

Mickevincent commented 8 years ago

I want it because it looks better than anything else I've seen. But if it really is impossible there is not much to do. Thanx for your answers.

inactive123 commented 8 years ago

These are screenshots of FF7 taken with the GL renderer set to 1x and with the NEDI shader preset selected.

http://imgur.com/a/WCNM2

Looks reasonably close to Waifu2x and at least it's rendering in real-time.

Mickevincent commented 8 years ago

Thanx, looks nice. I did this with some shaders in retroarch and reshade. The noise from the debanding gets very visable in pictures thou http://i.imgur.com/LGTynKw.png http://i.imgur.com/sIw1gPk.png http://i.imgur.com/XahR1ZA.jpg

Mickevincent commented 7 years ago

Since Waifu2x is so slow, it is not possible in real time. However, this is supposedly possible to run in real time quite easily and could be, as I understand, as good as waifu2x. So is it possible that anyone could make this into a scaler/shader for retroarch?

http://mmlab.ie.cuhk.edu.hk/projects/FSRCNN.html Codes and datasets are furthest down on the page.

Looking forward an answer =) /Mike

hizzlekizzle commented 7 years ago

It may be possible but it's not going to look like waifu2x since it's a different algorithm trained on a different dataset. I'll keep an eye on it, though, in case we can do something with it in the future.

We did just get a nice port of NNEDI3, which looks very good, if you want to check it out.

Mickevincent commented 7 years ago

Thanx for your answer hizzlekizzle. As I have understood it, it would be possible to train it to get the same look, or at least a similar. But yeah I have tried nnedi3. Sadly, then I prefer Super-Xbrz for pre-rendered games.