search

bjin / mpv-prescalers

prescalers for mpv, as user shaders
GNU Lesser General Public License v3.0
355 stars 34 forks source link

Upscaling Lower Resolution Video to 4K (Primer) #22

Closed classicjazz closed 6 years ago

classicjazz commented 6 years ago

@haasn, @bjin, and @wm4

I have been following your collective work on Vulkan (v2) and RAISR/RAVU since the summer. I have attempted to distill it into an article on my blog: "Upscaling Lower Resolution Video to 4K". Yours is amazing work though I am still trying to wrap my head around it.

1) I would appreciate your feedback if there are any mistakes or missing information in the primer. My specific focus is providing a multi-room 4K setup box solution using Nvidia Shield TVs, not a single room, Windows PC solution. Full disclosure: my experience to date has been with Kodi/SPMC. But, I know that MPV is embedded in Plex, which runs on the Shield.

I am a bit fuzzy on the exact application by source video resolution. Presumably, for a given broadcast/streamed/stored video:

1080p:

1080i:

720p:

480i:

I am not clear whether RAVU smooth is needed to avoid an additional anti-ringing function.

My assumptions are that:

Several additional questions:

2) have you done any performance testing in general to compare Vulcan-enabled RAVU to OpenGL EWA upscaling? I saw the NNEDI comparison numbers.

3) same question, specific to the Shield? Are there any performance tradeoffs / limitations (e.g. limited to 2x, limited to luma channel only, different radius--RAV1 v RAV5)?

Thanks!

classicjazz commented 6 years ago

@haasn, @bjin, and @wm4

I have been testing my assumptions above on my 2017 Retina 5k iMac using OpenGL. Admittedly, it is an imperfect proxy for 4K upscaling with the Nvidia Shield because it is 5k and it has a 4GB Radeon Pro 575 built in, with a much more powerful CPU. And, there seem to be a lot of issues with MPV on the Mac. (see https://github.com/mpv-player/mpv/pull/4833). My mpv.conf configuration file is described here: http://freetime.mikeconnelly.com/archives/5343

First, on the Mac, is there a way to force 1:1 pixel mapping of 3840×2160 output for testing purposes to better compare upscaling tradeoffs? Ideally, I would enable full screen and see 1:1 with letterboxing for the difference between 4k and 5k (5120 x 2880).

Currently, I see this in Terminal: VO: [opengl] 1920x1080 yuv420p

So, I assume the output on the iMac whether windowed or fullscreen is 1080 and something other than MPV is scaling the video. Similarly, I want to output from MPV in either YUV444 or RGB (else, I am reducing chroma by 1/2 and then something other than MPV is upscaling the chroma).

Following are my other observations:

1) MPV doesn't seem to apply resolution-specific settings like I described in my first message.

As I understand it, I need to globally apply the RAVU prescalers recursively in mpv.conf.

For example:

8x RGB Prescaling (to solve for 480i/DVD content upscaling)

opengl-shaders="~~/shaders/gather/ravu-r4-rgb.hook" opengl-shaders-append="~~/shaders/gather/ravu-r4-rgb.hook" opengl-shaders-append="~~/shaders/gather/ravu-r4-rgb.hook"

[note that your Wiki describes using "glsl-shaders-append="~~/shaders/ravu-r3.hook"", which I believe is Vulkan-specific since it throws an error using either profile=opengl-hq or profile=opengl-cb]

Since I want to upscale 480i, 720p and 1080i/p content to 2160p, I assume that I need to always invoke the RAVU preshader 3 times for all content. Then, I need to constrain this by using something like "max-downscaling-ratio=1.6". Assuming that is correct, is it really best both in terms of quality and performance to upscale of an upscale of an upscale? It would seem like that adds the potential for a lot of artifacts and is slower than a single resolution-specific prescaler (with a mapping like I described above)? Additionally, if you need to repeat the doubling 3x, then downscale, for embedded applications it would seem better to double 2x, then upscale with ewa_lanczossharp because this eliminates a step and saves processing time.

2) debanding is enabled by default in MPV though I haven't found optimal settings for 12bit Rec2020 output.

3) deinterlacing doesn't work automatically with MPEG2 (ATSC) video in MPG containers. You can force it for all videos. Even then, it doesn't do as good a job as the Nvidia Shield's deinterlacer particularly on news / sports tickers.

4) motion interpolation shouldn't be enabled if you plan to run the RAVU prescaler more than once. On the iMac, it leads to a lot of dropped frames. This is less of an issue for me because the UHD TV can always perform motion interpolation separately, if desired.

5) cscale=haasnsoft is broken. (see https://github.com/bjin/mpv-prescalers/wiki/Comparison) If I enable this, then all the colors are either B+W or significantly shifted. Switching back to ewa_lanczossharp solves this.

6) Upscaling of 720 and lower resolution content looks artificially flat. I can add noise through an additional filter. Like the upscalers, it appears like this is global setting applied regardless of the source video's resolution.

haasn commented 6 years ago

First, on the Mac, is there a way to force 1:1 pixel mapping of 3840×2160 output for testing purposes to better compare upscaling tradeoffs?

Disabling virtual resolution might help.

which I believe is Vulkan-specific

It isn't, you just need newer mpv

profile=opengl-hq or profile=opengl-cb

profile=opengl-hq is irrelevant. profile=opengl-cb makes no sense

Assuming that is correct, is it really best both in terms of quality and performance to upscale of an upscale of an upscale

It's the only way

cscale=haasnsoft is broken

OS X bug. I think it might be fixed in newer versions of mpv?

classicjazz commented 6 years ago

@haasn

Thanks for your suggestions.

On the iMac 5k, you can't truly disable scaling through the GUI. The best you can do is change your virtual resolution to 3200x1600. That's why I thought a full screen mode might work better for 1:1 testing.

I am using the latest build available through homebrew: mpv 0.27.0 (C) 2000-2017 mpv/MPlayer/mplayer2 projects built on Sun Oct 15 20:26:47 BST 2017

The opengl-cb was specified here: https://github.com/mpv-player/mpv/pull/4833 as a Mac workaround

I don't believe that I can test Vulkan on a Mac. I was hoping that to use the Mac to better understand how to use MPV on the Shield.

MPV is at the core of Plex. And, Plex is preloaded on the Shield. Are there settings that you are using for Plex for the Shield that you can share?

classicjazz commented 6 years ago

Adding no-hidpi-window-scale to mpv.conf appears to disable scaling on the iMac. But I still need to figure out how to force

VO: [opengl] 3840×2160 yuv444p

not:

VO: [opengl] 1920x1080 yuv420p

Otherwise, I don't believe that I can see the RAVU upscaling (because it is resized to 1080).

bjin commented 6 years ago

I don't get time to read the article throughly, just some quick note.

  1. You understanding about chroma upscaling (in mpv) is wrong. chroma is upscaled to luma resolution (video size), and then the converted RGB is upscaled to target resolution (screen size). There is no 2x luma + 4x chroma. At least not with current nnedi3 and RAVU.
  2. RAVU-Lite can be trained to perform 3x and 4x upscaling directly, I just haven't heard enough demand on this.
  3. 480i usually means NTSC/30fps, and usually requires IVTC instead of deinterlacer.
  4. You can use stats.lua (now intergrated) to verify that RAVU (one or multiple) is applied.

Most of your questions are regarding general mpv usage, and better ask on mpv IRC.

classicjazz commented 6 years ago

Thanks @bjin and @haasn for all your guidance.

I updated the primer. I also documented my configuration settings that automatically switch prescaler/upscaler profiles based on video resolution here. If you have a spare moment, I would appreciate your feedback.

I will keep an eye out for further Vulkan and RAVU development. For <=720 content, I think it would be interesting to have a single luma prescaler and a single chroma prescaler that is trained/influenced by the luma prescaler. It might have fewer artifacts and operate more quickly than recursive RAVU calls.

I was thrown by the "VO: [opengl] 1920x1080 yuv420p" when the windows geometry was set to "geometry=3840x2160". Using the -v switch confirms the 3840x2160 resolution is using RAVU upscaling. As a user, I would expect it to say "VO: [opengl] 3840×2160 RGB".

bjin commented 6 years ago

@classicjazz I implemented basic 3x upscaling variant of RAVU. It's currently compute shader only (thus under the compute folder), and unfortunately won't work on macOS. Feel free to try and provide feedbacks.

classicjazz commented 6 years ago

@bjin Interesting. I always thought prescalers had to be doublers. If I understand correctly, this would permit 720p video to be upscaled to 2160p in one pass. Also, it would permit 480i video to be sextupled in two passes, then downscaled to 2160.

You are right, the compute shader only displays a blue screen on the Mac. I will need to find another way to test, unless you plan to release a gather shader.

Here is my profiles in mpv.conf based on resolution. Not sure if they are correct.


[uhd]
# 7680x4320 (UHDTV)
# 3840x2160 (UHDTV)
profile-desc=cond:get('width', -math.huge) >= 3840
# No upscaling required

[full-hd]
# 1920x1080 (HDTV, Blu-ray)
profile-desc=cond:get('width', -math.huge) == 1920
# Video doubled by RAVU (1080 -> 2160); no additional up/downscaling
opengl-shaders="~~/shaders/gather/ravu-r4-yuv.hook"

[hd]
# 1280x720 (HDTV, Blu-ray)
profile-desc=cond:get('width', -math.huge) == 1280
# Video tripled by RAVU (720 -> 2160); no additional up/downscaling
opengl-shaders="~~/shaders/compute/ravu-3x-r4-yuv.hook"

[sdtv-ntsc]
# 640x480 (NTSC Broadcast)
# 704x480 (NTSC Broadcast)
# 720x480 (NTSC DVD)       
profile-desc=cond:get('height', -math.huge) == 480
deinterlace=yes
# Video sextupled by RAVU (480 -> 2880); then downscaled (2880 -> 2160)
opengl-shaders="~~/shaders/compute/ravu-3x-r4-yuv.hook"
opengl-shaders-append="~~/shaders/gather/ravu-r4-yuv.hook"
# Anti-Ringing
scale-antiring=0.7                      # Luma upscale deringing (Higher = Less ringing, but more detail loss)
dscale-antiring=0.7                     # Luma downscale deringing (Higher = Less ringing, but more detail loss)
cscale-antiring=0.7                     # Chroma upscale deringing (Higher = Less ringing, but more detail loss)

[sdtv-pal]
# 352x576 (PAL Broadcast)
# 480x576 (PAL Broadcast)
# 544x576 (PAL Broadcast)
# 720x576 (PAL Broadcast or DVD)
profile-desc=cond:get('height', -math.huge) == 576
deinterlace=yes
# Video quadrupled by RAVU (576 -> 2304); then downscaled (2304 -> 2160)
opengl-shaders="~~/shaders/gather/ravu-r4-yuv.hook"
opengl-shaders-append="~~/shaders/gather/ravu-r4-yuv.hook"
# Anti-Ringing
scale-antiring=0.7                      # Luma upscale deringing (Higher = Less ringing, but more detail loss)
dscale-antiring=0.7                     # Luma downscale deringing (Higher = Less ringing, but more detail loss)
cscale-antiring=0.7                     # Chroma upscale deringing (Higher = Less ringing, but more detail loss)
bjin commented 6 years ago

Your config seems correct. But keep in mind that all shaders comes with max-downscaling-ratio set to 1.414. Shaders will disable themselves if upscaling is unnecessary (require larger downscaling ratio after upscaling). For example, if your monitor is exactly UHD, you can combine uhd/fhd/hd profiles, and apply ravu-3x and ravu in turn. The ravu shaders will detect the video/screen size in the runtime.

I probably wont release a gather version of ravu-3x, it requires two seperate passes and will be about two times slower. Considering repeated use of ravu is already pretty fast, it don't worth the effect to have a slow ravu-3x