Add the NGU scaling algorithm which is in madvr to MPV?

[JulianLai]

mpv version and platform

mpv 0.21.0 on Linux

[haasn]

Patches welcome?

[kkkrackpot]

Personally, I would suggest neural scalers from here https://github.com/alexjc/neural-enhance :))) They're "said" to be better than madshi's NGU, and at least they're open and GPL-ed. But since personally I'm too stupid to provide any other help for this, I leave it all at anyone's discretion.

[ghost]

GPL = bad

[Hrxn]

Personally, I would suggest neural scalers from here https://github.com/alexjc/neural-enhance

What if you could increase the resolution of your photos using technology from CSI laboratories?

I WANT THAT!

They're "said" to be better than madshi's NGU, and at least they're open and GPL-ed.

Interesting, do you have a source for this by chance?

GPL = bad

Interesting, do you have a source for... Nevermind.

[mia-0]

Sigh. Yet another meme oil painting scaler with cherry-picked examples (i.e. low-res versions of the high-res samples the thing was trained on)?

[kkkrackpot]

@lachs0r if you're about that neural thing -- I don't know, sorry... It can be trained with various patterns, afaik. But this stuff is far beyond my knowledge, I could only suggest to contact its author directly.

@Hrxn if "source" means "where they say" it -- yes, I could point you to there, but I doubt it will be useful, because it's just yet another users forum, not even in a european language. On Doom9 there's a post with upscales comparision including NGU. I took the original photo there (that 50% downscaled with Catmull-Rom ), and upscaled it 2x with that neural thing with its default settings: https://0x0.st/LMw.png (comparision) Anyone interested can decide. BTW, I don't mind having NGU, too. :))

[paulguy]

Could just do what we do with NNEDI3 and have someone make them in to user shaders in a separate repo so people can just add them in and it won't be a licensing issue for mpv.

[oldie77]

@fhlfibh

They're "said" to be better than madshi's NGU

Done a quick comparison, using the following images from the Neural-Enhance pages:

https://github.com/alexjc/neural-enhance/issues/77 https://www.flickr.com/photos/benarent/3385036852/ https://www.flickr.com/photos/cyalex/10089086643/

Here are the results, comparing Neural-Enhance with NGU (with "soften edges" set to 2, and "add grain" set to 4), and with Jinc/EWA-Lanczos:

http://screenshotcomparison.com/comparison/193110 http://screenshotcomparison.com/comparison/193111 http://screenshotcomparison.com/comparison/193112 http://screenshotcomparison.com/comparison/193113

So which do you prefer? To me it looks like Neural-Enhance has a lot of aliasing and ringing problems.

@lachs0r

cherry-picked examples (i.e. low-res versions of the high-res samples the thing was trained on)

Really!? Anybody doing that? Sounds like cheating to me!! After a peek at the Neural-Enhance issue comments, it seems that Neural-Enhance is trained on the ImageNet database. The test images used to advertise Neural-Enhance's benefits don't seem to be part of the ImageNet database. That appears proper to me.

After a quick look at the Neural-Enhance NN configuration, it seems to have more than twice as many convolutional layers than waifu2x, and higher filter numbers, too. So I would guess Neural-Enhance is probably more than 2x slower than waifu2x, and waifu2x is already very slow. Just a quick guess, though, I haven't actually benchmarked any of this.

[haasn]

So which do you prefer? To me it looks like Neural-Enhance has a lot of aliasing and ringing problems.

Same. Of those screenshots I roughly prefer High-Res > Jinc > NGU > Neural-Enhance.

[ArchangeGabriel]

Jinc looks poor to me compared to NGU/Neural-Enhance. Like very blurry. Not sure about NGU vs Neural-Enhance, the last one looks more crisp but I suppose this is because of aliasing and ringing effect. Anyway, best is High-Res. :p

[Hrxn]

Agreed. High-Res is the best scaler!

[haasn]

Agreed. High-Res is the best scaler!

Seems the consesus is reached, we'll add --prescale=highres in the next release.

[Hrxn]

+1

It's about damn time to settle this issue once and for all!

[kkkrackpot]

@oldie77 not sure of what you did with that images exactly, but out of those four I'd prefer the Hi-res one.

[oldie77]

The Neural-Enhance dev took a High-Res image, downscaled it, and then upscaled it again with Neural-Enhance. This is the usual approach, so that the Neural-Enhance upscaled image can be compared to the original High-Res image, in order to get an objective measurement of how good the upscale really is. Obviously, the closer any upscaled image comes to the original High-Res image, the better the upscaled image should be considered to be.

The image I named "High-Res" is of course the original image before downscaling. I've added this image so we can all compare which upscaled image comes closest to the original. haasn is just being funny. :)

Again, to be clear: The NGU, Neural-Enhance and Jinc images in my screenshot comparison are upscaled images. The High-Res image is not upscaled, it's the original image before down+upscaling.

[haasn]

Obviously, the closer any upscaled image comes to the original High-Res image, the better the upscaled image should be considered to be.

(At reversing that particular downscaling algorithm)

[oldie77]

Quite true. Some algos might "cheat" by just learning how to undo one specific downscale operation, instead of learning how to upscale any image.

[kkkrackpot]

@oldie77 Hmmm, actually personally I doubt that such down->up comparision method meets our real world conditions, i.e. what mpv does when upscaling. Anyway, considering NGU vs Neural here http://screenshotcomparison.com/comparison/193112 -- I see Neural being sharper, with more details, although with more "ringing artefacts". PS. Strangely, it's kinda opposite here http://screenshotcomparison.com/comparison/193110

[haasn]

I wonder how well throwing a deconvolution at the people who like these over-sharpened, ringing meme filters would work.

The results can look pretty similar: http://yuzhikov.com/articles/BlurredImagesRestoration1.htm

[oldie77]

@fhlfibh

I doubt that such down->up comparision method meets our real world conditions

Fair enough. Of course in the real world we don't have a "High-Res" image to compare to. However, if we want to objectively evaluate the quality of an upscaling algorithm, it does make sense to have something to compare to, because otherwise it's all just totally subjective, so this down->up approach generally seems like a good idea to me. But to make it proper, we should probably choose the downscaling algorithm randomly, or something like that, to make sure none of the upscaling algos can cheat by optimizing for just one specific downscaling algo.

@fhlfibh

I see Neural being sharper, with more details

To be brutally frank, I think what you see as sharper / more detail is to a certain extent just artifacts. And what is worse: In motion some of these artifacts would probably not be stable, but change from frame to frame.

Here's a blown up (using point filtering) comparison to make things easier to see:

https://s17.postimg.org/9v2ktncb3/street_Comparison.png

The order from top to bottom is: High-Res, NGU, Neural-Enhance, Jinc.

@haasn, might be worth a try. Do you happen to know some website where we can run images through deconvolution for a quick comparison test?

[Hrxn]

The image I named "High-Res" is of course the original image before downscaling. I've added this image so we can all compare which upscaled image comes closest to the original. haasn is just being funny. :)

Oh, really?!

Personally, I'd prefer NGU from the given examples, but I'm no fan of screenshot comparisons for that specific purpose, I need to see it in motion to see how "distracting" artifacts really are.

Agree with the point raised about downscaling, that is hard enough to get "right" (if at all), so any conclusions about upscaling afterwards are a bit questionable. Keep in mind that this also is a destructive operation.

[lvml]

Here's another candidate algorithm for upscaling: RAISR: Rapid and Accurate Image Super Resolution which is advertised a lot by Google, and claimed to be faster than other state-of-the-art upscalers.

[wiiaboo]

Yet another neural network upscaler.

[CounterPillow]

Do these even run in the same order of magnitude when it comes to time compared to scalers that mpv uses? Also, since neural networks have to be trained, I'm assuming all the pre-trained weights would have to be shipped with it?

[Hrxn]

[..] for my own use case (watching anime on hidpi screen with low-end graphics card). [..]

Why am I not surprised?

[lvml]

On 08/10/2017 09:19 AM, bjin wrote:

You can find the shaders here https://github.com/bjin/mpv-prescalers/tree/master#about-ravu

For those who would like to use such shaders for upscaling but have no pre-existing experience which of the many shader names corresponds with what kind of result it would be great if you could provide a sample screen-shot (original / upscaled) for each shader in your github project.