Anime4K_Upscale_GAN_x2_M seem to be producing some minor artifacts that are not present in the other GAN variants

[Z-Dante]

Not really sure if this is a bug or not, but earlier I was playing with the new GAN shaders and noticed one thing. The X2_M variant of the GAN shader seem to be producing some artifacts here and there, which are not present in any other GAN shaders. To demonstrate, here are some images.

1. Some noisy old gameplay record 480p->1080p: Zoom in and look at the border of the buttons if you can't notice the artifacts
2. 720p -> 1080p. 400% zoomed view: And album of the full images for reference: https://imgur.com/a/qV66BA3

All tested with the latest windows build of MPV on default settings on a 1080p display(I don't have 4k :-x ), running on RTX2060 GPU.
I might be nitpicking but feels odd to me that only one variant of the GAN shaders would produce such artifacts.

And good work on the new shaders. These seem to be much better at keeping grainy textures intact compared to the CNN ones.

Edit: My input.conf:

CTRL+1 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders//Anime4K_Upscale_CNN_x2_L.glsl"; show-text "Anime4K: Upscale L CNN"
CTRL+2 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Upscale_GAN_x2_S.glsl"; show-text "GAN X2 S"
CTRL+3 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Upscale_GAN_x2_M.glsl"; show-text "GAN X2 M"
CTRL+4 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Upscale_GAN_x3_L.glsl"; show-text "GAN X3 L"
CTRL+5 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Upscale_GAN_x3_VL.glsl"; show-text "GAN X3 VL"
CTRL+6 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Upscale_GAN_x4_UL.glsl"; show-text "GAN X4 UL"
CTRL+7 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Upscale_CNN_x2_VL.glsl"; show-text "CNN VL"
CTRL+0 no-osd change-list glsl-shaders clr ""; show-text "GLSL shaders cleared"```

[bloc97]

Thanks for the in-depth comparison pictures! I think that it's simply by chance the M shader didn't converge correctly during training. I will try to fix this before the official release of shaders.

[bloc97]

In fact I think I can see very tiny (1 pixel) artifacts in the L version too. There might be broader convergence issues from the adversarial training. I will have to investigate. Solving this correctly might take more time as there's no papers that tackle training tiny SRGANs. For now I will try re-training both networks to see if the issue repeats itself.

[hooke007]

By my test, upscale GAN x2-UL could produce the best quality while the least artifacts, but it was removed. @bloc97 Could we get another choice to replace GAN x2-UL later? x2-m x2-UL x3-L x3-VL x4-UL

[bloc97]

That's because GAN x2-UL was trained with small amounts of denoising, I will see if we should support 6 GAN+Denoise upscalers, or let the GAN-Restore handle denoising.

[bloc97]

The optimal case would be to delegate all degradations to RestoreGAN shaders, and let the SRGAN work with pristine images. The current problem is that Restore shaders sometimes introduce ringing, blurs out details or do not restore lines enough.

This happens because the restore shader has no idea of the whole image and only works on small patches of the image. It cannot know if the entire image is blurred or whether the blur was intentional and only for a part of the image. This can cause the restore shader to try removing noise or blur where there is no degradation and cause a loss of detail/ringing.

A lazy method was used for the v4.0 restore shaders, by training one shader (non-soft version) on some degradation and another (soft version) on other types of degradation.

By implementing global feature-wise transformations we can hope the new RestoreGAN shaders will not denoise, deblur or add ringing to images that have no degradation.

[Z-Dante]

I went back to check the old GAN x2-UL shader and it does seem to be working great and producing less artifacts for most cases. But it also has some edge cases where it was losing a lot of the minor details on grainy textures (not as much as the CNN shaders but still). Would be great if there was a way to reduce the artifacts while minimizing the loss of details ^^ https://imgsli.com/ODQyMjU/0/1

[bloc97]

That's basically what we're trying to do with feature-wise transformations. If the shader can actually know how much compression is in an entire image, it can learn to not remove details when overall noise is low.

Compression and degradation is almost always applied to the entire image, while artistic degradation is usually applied to parts of an image. The important thing is to distinguish between the two by giving the neural network a way to look at the entire image without slowing it down too much.

[bloc97]

Upon further examination of the image you provided, it is a very good example of having all kinds of artistic degradation + unintended noise from compression. There are four things:

1. The gaussian noise (film-grain like noise) should not be removed as it is artistic and intended (you can look at the blurry parts of the image at the top and bottom, they don't have noise) Current Restore shaders would definitively try to remove it, making the image lose detail.
2. The blur at sides of the images (vignette) is artistic and should not be removed. Current restore shaders will also try to remove it, and wrongly oversharpening parts of the image, causing ringing artifacts.
3. There doesn't seem to have chroma subsampling, current restore shaders always assume 4:2:0 chroma subsampling and will cause color artifacts.
4. There is JPEG/MPEG (more commonly, DCT) compression, this should be removed as usual, current restore shaders always assume a moderate amount of compression, which does the right thing here.

If you try restore shaders with this image it will fail considerably, as expected. The goal is for the new RestoreGAN shaders to take in account all four cases, so that it can make a better restoration overall.

[bloc97]

If the RestoreGANs will be successful at restoring an image without distorting it, the SRGAN shaders will not need to denoise the image. This makes both training and using the shaders much easier, and we would not need a second set of SRGAN+Denoise shaders:

Pristine images from pixiv: Use SRGAN only. Visual novels: Use SRGAN only. Degraded images from web: Use RestoreGAN + SRGAN. Anime: Use RestoreGAN + SRGAN.