[FR] Image scaling algorithms and sharpening

radry commented 10 months ago

Describe your suggested feature

Often Manga scanlations have a lower resolution than modern phone and tablet screens, which means they will get scaled up while displaying them. This results in a blurry image, although slightly.

Modern devices should have enough power to support a better upscaling algorithm (for example Lanczos?) than bilinear and/or to apply a (customizeable) real time sharpening filter.

Going forward also local AI upscaling should be considered as a long term goal. Even current devices already have NPUs for accelerated AI applications, so it's not impossible.

tl;dr:

1) Better image upscaler than bilinear
2) Customizeable (real time) sharpening filter 3) Long term prospect: AI upscaling (ESRGAN or better/newer with suitable manga model), for example as postprocessing after automatic download.

Other details

No response

Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] If this is an issue with an official extension, I should be opening an issue in the extensions repository.
[X] I have updated the app to version 0.15.3.
[X] I will fill out all of the requested information in this form.

AGuyNamedRakkun commented 10 months ago

Adding to this, here is a comparison between Tachiyomi's (default scaling which I think Mihon uses too) and Perfect Viewer (which uses Lanczos scaling)

radry commented 10 months ago

After digging through the code, it seems it uses davemorrissey/subsampling-scale-image-view library to display the images. So it probably depends on the functionality of that library what can be done in this app. I open an issue there to see if they bother to add it in their library.

AGuyNamedRakkun commented 10 months ago

The last commit made in that repository was 4 years ago. I don't think the developers/maintainers will even read that issue. The possibilities are either you could rewrite it yourselves or wait for some contributor to pick up this issue and hopefully submit a PR (for which the chances are highly unlikely).

srappan commented 7 months ago

https://www.imagemagick.org/Usage/filter/

I would recommend looking at imagemagick docs to learn about scalers TL;DR for who don't want to read theory Mitchell for downscaling manga and the like to avoid moire artifacts and preserve screen tones.

Ewa_lanczos or ewa_hanning for upscaling default.

MajorTanya commented 7 months ago

https://www.imagemagick.org/Usage/filter/

I would recommend looking at imagemagick docs to learn about scalers TL;DR for who don't want to read theory Mitchell for downscaling manga and the like to avoid moire artifacts and preserve screen tones.

Ewa_lanczos or ewa_hanning for upscaling default.

That's a CLI tool though isn't it, I don't see a way to integrate that into Mihon in a sensible way, be it as a CLI tool or as bundled native libraries written out in C or something.

srappan commented 7 months ago

I am aware, I was just mentioning the docs as useful for the theory behind image scalers, it's just math that's needs to be implemented, either using a existing library or written from scratch

BrutuZ commented 7 months ago

Basically a port of Tachiyomi/#10099 🤔

MajorTanya commented 7 months ago

I am aware, I was just mentioning the docs as useful for the theory behind image scalers, it's just math that's needs to be implemented, either using a existing library or written from scratch

I think you severely underestimate what you're asking if this is how you describe it. Yes, it's "just math" the same way particle physics is "just smashing things together". These algorithms would have to be implemented in a more low level language like C or Rust to be even remotely fast. This is a Kotlin project and an Android app. Not saying it's impossible or won't happen maybe eventually, but it's just someone putting

if (imageTooBig) {
  doTheLanczos();
}

in the code and we're done.

radry commented 7 months ago

This isn't about DOWNscaling but UPscaling...

Xori71 commented 7 months ago

This isn't about DOWNscaling but UPscaling...

~~No, I think it’s downscaling. Very high resolution images have dimensions that are well beyond what any phone offers. Thus, we get the moire effect.~~

Nevermind. When I replied, for some reason I had the idea that your post was about images that were higher resolution than the display. My bad.

MajorTanya commented 7 months ago

It's about both, since the same option would be sensible to offer for downscaling too.

AGuyNamedRakkun commented 7 months ago

These algorithms would have to be implemented in a more low level language like C or Rust to be even remotely fast.

There is a library in cpp for this but it has to be ported to an AAR library.

Shana-V commented 7 months ago

waifu2x works very well，Maybe refer to this program https://github.com/ArchieMeng/ncnn-android-waifu2x-demo

mutsumi63 commented 7 months ago

About Upscale https://github.com/ArchieMeng/ncnn-android-waifu2x-demo You can check out this demo about upscale. It uses waifu2x-ncnn-vulkan, and it runs quite well on android phone with GPU

MajorTanya commented 7 months ago

About Upscale https://github.com/ArchieMeng/ncnn-android-waifu2x-demo You can check out this demo about upscale. It uses waifu2x-ncnn-vulkan, and it runs quite well on android phone with GPU

What are the app size and performance implications of this, especially on lower end devices? Consider that Mihon can be run on an Android 8 e-reader

Xori71 commented 7 months ago

About Upscale https://github.com/ArchieMeng/ncnn-android-waifu2x-demo You can check out this demo about upscale. It uses waifu2x-ncnn-vulkan, and it runs quite well on android phone with GPU

I’ll have to be honest here, AI upscaling doesn’t excite me. The final product is basically a water color painting, and when it isn’t, that basically means that the source image was of already high enough quality.

I’d much rather look at a somewhat blurry image with coherent lines than a mush of ink.

MajorTanya commented 7 months ago

Upscaling is much less of a concern for Mihon anyway, since the downsides of low quality images are not its problem really. But if there is an avenue to get downscaling of some description into Mihon, it would enable Mihon to mitigate/all of the Moiré effects of super-high quality images.

Xori71 commented 7 months ago

Upscaling is much less of a concern for Mihon anyway, since the downsides of low quality images are not its problem really. But if there is an avenue to get downscaling of some description into Mihon, it would enable Mihon to mitigate/all of the Moiré effects of super-high quality images.

100% agree. Some form of downscaling would be amazing.

mutsumi63 commented 7 months ago

What are the app size and performance implications of this, especially on lower end devices? Consider that Mihon can be run on an Android 8 e-reader

Need a device with Snapdragon 835 or higher. Program size will increase by tens of MB. It could be made into an optional feature or an expansion pack, so it won't affect low-performance devices and app size.

I just wanted to express that some people do need upscale. Because some manga sources have very low resolution, It would bring some improvement.

waifu2x-ncnn-vulkan is actually very suitable for running on current phones. For those who want to test the performance of waifu2x upscale on phone, can search "waifu2x ncnn" on GP. The author of that demo has created a nice app.

Xori71 commented 7 months ago

What are the app size and performance implications of this, especially on lower end devices? Consider that Mihon can be run on an Android 8 e-reader

Need a device with Snapdragon 835 or higher. Program size will increase by tens of MB. It could be made into an optional feature or an expansion pack, so it won't affect low-performance devices and app size.

I just wanted to express that some people do need upscale. Because some manga sources have very low resolution, It would bring some improvement.

waifu2x-ncnn-vulkan is actually very suitable for running on current phones. For those who want to test the performance of waifu2x upscale on phone, can search "waifu2x ncnn" on GP. The author of that demo has created a nice app.

Very low resolution and AI upscaling is not always the best combination. Forget the art, letters would be unintelligible, due to their small size and post-upscaling artifacts.

Although I can see it being maybe useful for mid-res images.

MajorTanya commented 7 months ago

As stated before, sources using low quality images is not something Mihon has to concern itself with. When users need higher quality images, they should find a different source that provides higher quality content or something.

Moiré pattern interference is something related to how Mihon itself displays very-high quality images, that's why I keep pointing to downscaling.

EnArvy commented 5 months ago

https://github.com/ser-gik/smoothrescale

This appears to have all the relevant info to implement lanczos using libswscale from ffmpeg

realFPS commented 3 months ago

https://github.com/T8RIN/ImageToolbox

Maybe this can help as reference. This app has many scaling modes for resizing images, it also has different versions of lanczos.

I use Perfect Viewer for reading digital volumes with lanczos 3 and the results are drastically better for high resolution images compared to mihon. Please consider this, If mihon gets better at scaling images, it will become the all in one package for manga anyone would ever need for android.

MajorTanya commented 3 months ago

https://github.com/T8RIN/ImageToolbox

Maybe this can help as reference. This app has many scaling modes for resizing images, it also has different versions of lanczos.

I use Perfect Viewer for reading digital volumes with lanczos 3 and the results are drastically better for high resolution images compared to mihon. Please consider this, If mihon gets better at scaling images, it will become the all in one package for manga anyone would ever need for android.

For reference, that app is using Aire. Aire is bundling C++ code and appears to be Android-specific or at least largely so.

jack-mil commented 2 months ago

This should be a very high priority. Proper resampling methods need to be available, particularly because down-sampling manga is very prone to moiré due to screentone patterns. It is very common for digital manga sources to have images much larger than the resolution of most displays. I know DSP, but I'm not a Kotlin developer, so I can't help out in that regard.

AI is completely unnecessary, this is just a matter of using higher quality resampling algorithms implementations. Surely there is some API available at some level for Android developers who need to resize textures before drawing to the screen?

AntsyLich commented 2 months ago

If anyone wants to help out on rewriting the image decider with https://github.com/libvips/libvips (which has resampling support) contact us @ discord. And since no one has anything productive to share I'll lock this thread.

mihonapp / mihon