v4.x Roadmap

[bloc97]

New features concerning v4 will be added here.

v4.0.x

• [x] Implement restore shaders for 1080p anime.

v4.1.x

• [x] General exploration on whether using GANs can improve real-time upscalers. (Yes they can...)

• [x] Stable training of GANs for real-time upscaling. (Successfully training a very small network using an adversarial utility function is hard.)

  • [ ] Explore ideas found in "The Contextual Loss for Image Transformation with Non-Aligned Data". (Better style regularization for the GAN, should improve stability and prevent mode collapse.)

  • [x] Explore ideas found in "Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis". (Faster training of GANs for quicker prototyping.)

  • [x] Explore ideas found in "Real-ESRGAN". (On first view their methods are very similar to what is already used to train the Anime4K restore networks. Maybe they can be improved to train lightweight ESRGANs...)

  • [x] Explore ideas found in "Compressing GANs using Knowledge Distillation". (Training a large GAN or using existing ESRGANs then pruning its size using knowledge distillation?)

  • [ ] Other ideas (feel free to propose any new idea below)

• [x] Implement GANs for upscale shaders. (Should be better than previous shaders to be considered done.)
• [x] Reduce aliasing and phantom textures in SRGAN.
• [x] Reduce size and implement smaller variants for SRGAN. (If possible)
• [ ] Investigate convergence issues for smaller SRGAN versions (M and L)
  • [ ] Fix #146
  • [ ] Investigate possible discriminator collapse
• [x] Implement restore shaders for 720p/480p anime. (Probably using GANs)
• [ ] Reduce size and implement smaller variants for RestoreGAN. (If possible)
• [ ] Reduce oversharpening in RestoreGAN.
• [ ] Reduce color shift in RestoreGAN. (undesired tonemapping?)
• [ ] Implement global feature-wise transformations for RestoreGAN.

v4.2.x

• [ ] Implement restore shaders for older pre-digital era anime. (Probably using GANs)
• [ ] Investigate better feature-wise transformations/attention networks for anime restoration.
• TBD

Other stuff (might take longer)

• [ ] Cleanup and release full training code for restore shaders.
• [ ] Cleanup and release full training code for GANs.
• [ ] Update the wiki.

[hooke007]

restore_GAN_UL will obviously change some color (restore_GAN_UUL would too but not so obvious) To watch the face and the background (left part).

[bloc97]

This is one of many problems when using an adversarial loss. It's probably due to the discriminator not having seen some colors in the dataset. I will see if I can fix it before the release version.

[UltraHQ]

Will there ever be an update again?

[bloc97]

Yes, it's just that right now the implementation of many modern network architectures (eg. transformers, self-attention, etc.) require more advanced features that mpv currently does not support. They can be implemented by brute force in GLSL but it would be way too slow to be worthwhile. If you have a very high end CUDA-enabled GPU like the RTX 3090, there are many other upscalers that you can use that have better quality that Anime4K. There's currently a big update for libplacebo on the way that promises many new features that Anime4K can leverage.

[UltraHQ]

Yes, it's just that right now the implementation of many modern network architectures (eg. transformers, self-attention, etc.) require more advanced features that mpv currently does not support. They can be implemented by brute force in GLSL but it would be way too slow to be worthwhile. If you have a very high end CUDA-enabled GPU like the RTX 3090, there are many other upscalers that you can use that have better quality that Anime4K. There's currently a big update for libplacebo on the way that promises many new features that Anime4K can leverage.

Do you mind naming some? I do have an CUDA-enabled GPU, but every other upscaler I tested does not look as great as Anime4K.

[bloc97]

I think this is what I remembered seeing one time, but I haven't had time to try it out... https://github.com/the-database/mpv-upscale-2x_animejanai

[4NXIE7Y]

New features concerning v4 will be added here.

v4.0.x

• [x] Implement restore shaders for 1080p anime.

v4.1.x

• [x] General exploration on whether using GANs can improve real-time upscalers. (Yes they can...)

• [x] Stable training of GANs for real-time upscaling. (Successfully training a very small network using an adversarial utility function is hard.)

  • [ ] Explore ideas found in "The Contextual Loss for Image Transformation with Non-Aligned Data". (Better style regularization for the GAN, should improve stability and prevent mode collapse.)
  • [x] Explore ideas found in "Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis". (Faster training of GANs for quicker prototyping.)
  • [x] Explore ideas found in "Real-ESRGAN". (On first view their methods are very similar to what is already used to train the Anime4K restore networks. Maybe they can be improved to train lightweight ESRGANs...)
  • [x] Explore ideas found in "Compressing GANs using Knowledge Distillation". (Training a large GAN or using existing ESRGANs then pruning its size using knowledge distillation?)
  • [ ] Other ideas (feel free to propose any new idea below)
• [x] Implement GANs for upscale shaders. (Should be better than previous shaders to be considered done.)
• [x] Reduce aliasing and phantom textures in SRGAN.
• [x] Reduce size and implement smaller variants for SRGAN. (If possible)

• [ ] Investigate convergence issues for smaller SRGAN versions (M and L)

  • [ ] Fix Anime4K_Upscale_GAN_x2_M seem to be producing some minor artifacts that are not present in the other GAN variants #146
  • [ ] Investigate possible discriminator collapse
• [x] Implement restore shaders for 720p/480p anime. (Probably using GANs)
• [ ] Reduce size and implement smaller variants for RestoreGAN. (If possible)
• [ ] Reduce oversharpening in RestoreGAN.
• [ ] Reduce color shift in RestoreGAN. (undesired tonemapping?)
• [ ] Implement global feature-wise transformations for RestoreGAN.

v4.2.x

• [ ] Implement restore shaders for older pre-digital era anime. (Probably using GANs)
• [ ] Investigate better feature-wise transformations/attention networks for anime restoration.
• TBD

Other stuff (might take longer)

• [ ] Cleanup and release full training code for restore shaders.
• [ ] Cleanup and release full training code for GANs.
• [ ] Update the wiki.

Hey loved your work on this so far. Will be using it a bit for my undergrad project. Wanted to know if there has been any progress on:

• Cleanup and release full training code for restore shaders.
• Cleanup and release full training code for GANs.

[John-Gee]

Yes, it's just that right now the implementation of many modern network architectures (eg. transformers, self-attention, etc.) require more advanced features that mpv currently does not support. They can be implemented by brute force in GLSL but it would be way too slow to be worthwhile. If you have a very high end CUDA-enabled GPU like the RTX 3090, there are many other upscalers that you can use that have better quality that Anime4K. There's currently a big update for libplacebo on the way that promises many new features that Anime4K can leverage.

Is the current libplacebo enough or still not? Thanks a lot!

[srk24]

Thanks for the past work! Now will GAN be updated to fix the off-color issue?