Add larger GANs

[rafaelferreiranvr]

Could you please add larger GANs? I use a second GPU to do upscaling and i would like to use it while rendering at lower resolution At the moment i am using Anime4K_Upscale_GAN_x2_S to upscale 648p to 1080p, looks good but sometimes it fails with high frequency details, producing noise. Larger GANs would be perfect for it, like these: https://github.com/bloc97/Anime4K/blob/master/glsl/Upscale/Anime4K_Upscale_GAN_x3_L.glsl https://github.com/bloc97/Anime4K/blob/master/glsl/Upscale/Anime4K_Upscale_GAN_x3_VL.glsl https://github.com/bloc97/Anime4K/blob/master/glsl/Upscale/Anime4K_Upscale_GAN_x4_UL.glsl https://github.com/bloc97/Anime4K/blob/master/glsl/Upscale/Anime4K_Upscale_GAN_x4_UUL.glsl The last one can upscale 360p to 4k

[hooke007]

dup https://github.com/Blinue/Magpie/discussions/181#discussioncomment-1861197

[rafaelferreiranvr]

dup #181 (reply in thread)

So the GAN 2X UL would work, right?

[hooke007]

It hasn't been implemented.

[Blinue]

I won't port Anime4K_Upscale_GAN_x2_UL since it's too complicated. Still waiting for bloc97 to improve these shaders.

[bloc97]

The UL version is way too large to be ported manually, I'm not sure if there's a lot of demand for using such large shaders in magpie but I might implement in the future something to allow directly generating magpie shaders from tensorflow. Edit: By future I mean at least 6 months while I'm finishing my thesis. I don't have any free (literally) time to work on Anime4K at the moment...

[rafaelferreiranvr]

I won't port Anime4K_Upscale_GAN_x2_UL since it's too complicated. Still waiting for bloc97 to improve these shaders.

The UL version is way too large to be ported manually, I'm not sure if there's a lot of demand for using such large shaders in magpie but I might implement in the future something to allow directly generating magpie shaders from tensorflow. Edit: By future I mean at least 6 months while I'm finishing my thesis. I don't have any free (literally) time to work on Anime4K at the moment...

What are the steps envolved in porting a shader to magpie? Also, why magpie has its own way of shaders, it could'nt use GLSL directly? If most of the work consists in converting GLSL to HLSL, there's a tool that can help, it is called ShaderMan It was made to convert Shadertoy shaders so it can be used in Unity Link: https://github.com/smkplus/ShaderMan

[bloc97]

So the thing is normally, each shader is supposed to be standalone (eg. do one pass on the image only). However neural networks need multiple passes over the same image. Usually that means tens or hundred's of independent shader files that have to be loaded in a specific order.

What mpv and magpie's API allows you to do is to merge all the shaders in one and specify which ones to run, where to run and which textures to use.

Magpie's API (uses HLSL) is slightly more advanced and has more features than mpv (uses GLSL), which breaks compatibility even more. This was done probably to support a wider range of shaders.

To port the shader you have to first convert the shaders themselves from GLSL to HLSL, then convert all the "defines" which are different for Magpie and mpv.

At that point we could just generate HLSL shaders with magpie defines in the first place... Though someone could write a generalized converter that automatically translates mpv's shaders into magpie, but it would not be easy...

[rafaelferreiranvr]

So the thing is normally, each shader is supposed to be standalone (eg. do one pass on the image only). However neural networks need multiple passes over the same image. Usually that means tens or hundred's of independent shader files that have to be loaded in a specific order.

What mpv and magpie's API allows you to do is to merge all the shaders in one and specify which ones to run, where to run and which textures to use.

Magpie's API (uses HLSL) is slightly more advanced and has more features than mpv (uses GLSL), which breaks compatibility even more. This was done probably to support a wider range of shaders.

To port the shader you have to first convert the shaders themselves from GLSL to HLSL, then convert all the "defines" which are different for Magpie and mpv.

At that point we could just generate HLSL shaders with magpie defines in the first place... Though someone could write a generalized converter that automatically translates mpv's shaders into magpie, but it would not be easy...

If i successfully convert Anime4K_Upscale_GAN_x3_L.glsl for instance to HLSL, converting the defines would be easier? I mean, would it be easier to convert it to magpie?

[bloc97]

mpv's glsl is not a vanilla glsl, it uses special defines to pass through information otherwise unavailable to each shader stage. They're like glsl uniforms but are active only for a portion of the shader, if that makes sense.

Magpie does the same thing, thus you can't really use a glsl->hlsl converter, what you need is a mpv->magpie converter...

[rafaelferreiranvr]

mpv's glsl is not a vanilla glsl, it uses special defines to pass through information otherwise unavailable to each shader stage. They're like glsl uniforms but are active only for a portion of the shader, if that makes sense.

Magpie does the same thing, thus you can't really use a glsl->hlsl converter, what you need is a mpv->magpie converter...

This seems more dificult than i thought I'll study more about it so i can help you guys

[Blinue]

The UL version is way too large to be ported manually, I'm not sure if there's a lot of demand for using such large shaders in magpie but I might implement in the future something to allow directly generating magpie shaders from tensorflow.

It would be nice if this was possible. I'm trying to improve the performance of the Anime4K shaders, which can be used as a template. Thanks again for your excellent work, the Anime4K family is one of the most commonly used shaders in Magpie.

This seems more dificult than i thought I'll study more about it so i can help you guys

Yes, it's hard, so every little bit of help counts.