pixel shader support

[oct9]

any chance of adding pixel shader support? one of the few things missing from this amazing work (except for ability to use a front end a la smplayer).......

[grigorig]

What exactly do you expect? Some kind of support for custom filtering and scaling?

[ghost]

vo_opengl could easily support this. But the main question is: how should user defined pixel shaders be integrated? OpenGL shaders are a complex environment, so it's not always clear what to do.

Can you give some concrete examples? Not just use cases, but real world shader examples as text would be helpful too.

[OrdinaryMagician]

Oh I'm really interested in this feature. Support for custom GLSL shading would be a nice addition for my... experiments.

[Themaister]

Are pixel shaders expected to handle YUV -> RGB conversion themselves? Having the GL backend pre-converting to RGB via an FBO would make pixel shader part much simpler at least.

[ghost]

That's exactly the question. What inputs should the pixel shader take, what outputs...

[OrdinaryMagician]

I'm only experienced in GLSL 1.20, so I'm not familiar with the big changes in later versions.

Pretty sure there's enough with having just a sampler for the screen texture, then 2D vectors for the screen resolution and current texture coordinate, and possibly also a float providing the current time in ms. With only that, I'd have enough for most of the filters I want to implement.

[Themaister]

I've implemented something similar for RetroArch (game/emu interface) before. It's obviously RGB only so that case is a bit simpler, but the basics should transfer nicely to mpv.

For single pass shaders, I'd suggest something like this:

Vertex shader:

attribute vec2 VertexCoord; // clipspace x/y
uniform mat4 MVPMatrix; // If needed to support rotation and whatnot.
attribute vec2 TexCoord[3]; // tex coords for YUV. Allows supporting different sampling phase for subsampled chroma easily.

Pixel shader:

uniform sampler2D Texture[3]; // Potentially for planar YUV.
uniform mat4 YUVRGBMatrix;
uniform vec2 TextureSize[3]; // Support subsampling.
uniform vec2 OutputSize; // Viewport size
uniform vec2 InputSize[3]; // In case you need to support POT-only GL implementations where you pad the texture.
uniform float Time; // if desired

Branching inside the shader on YUV or RGB input is silly, so you could set a define from glShaderSource. Recompile the shader if input type changes. That makes writing shaders a bit more tedious though, but there might be shaders that want to scale luma and chroma differently.

Another question is whether or not you want to support the range of GLES2+, GL 2.x and GL 3.x+ core with a single "spec". That is possible, but requires some cooperation between the shader and GL code.

GLES 2.0 requires in pixel shader

#ifdef GL_ES
precision mediump float; // Or highp if GLES3+ or some particular define is set.
#endif

GL 3+ core requires

#version 140 (or whatever GL version you use)

as preamble, and attribute/varying are replaced with more generic in/out qualifiers, texture2D() is also renamed to just texture().

[ghost]

That's all pretty fragile. Maybe you could organize the C code in a way it doesn't need much code to support external shaders this way, but as soon as you change some internals, you'll end up with the need for a compatibility layer for user shaders.

Unless it's ok that user shader don't have a guarantee, and break every once in a while.

Not sure what's the alternative. Maybe one could restrict shader scripts to expressions that work on GLSL variables, instead of allowing user shaders to sample from video textures directly?

[Themaister]

If a shader cannot sample the texture how it wants, it's almost useless from a shader writers perspective though. Doubt you can avoid full GLSL without restricting it to the most basic shaders.

Then again, most of the complexities in the proposed interface comes from supporting both YUV and RGB. If the YUV->RGB conversion is "fixed" function it'd make things way easier.

EDIT: Looking at the "megashader" in video/out, it's probably a good idea to let mpv deal with YUV ...

[ghost]

I'll probably replace this "megashader" with GLSL generated in C directly. This megashader is actually a nice way to handle all the different stages and their combinations with somewhat uniform code (e.g. applying gamma correction, gamma un-correction, color management, color matrix, and that in combination with 1 pass for unscaled RGB, 2 passes for e.g. separated lanczos scaling of RGB, 3 passes for separated lanczos scaling with preceding RGB conversion pass...)

But yes, the processing pipeline is a bit littered with special casing to cater to color nerds. That's why it would require a well defined interface for shaders in some way or other - either that, or just provide "best effort" shader support, where we don't guarantee much other than "it works with this mpv release with this kind of video".

[oct9]

i'm not a programmer, just a htpc freak and i'm very impressed with the quality of mpv - looks as good as anything i play on mpc-hc (plus madvr, reclock, lav) and with less of a system footprint (using a very low-end nvidia gpu) - the only thing i'm missing is pixel shaders - currently using vibrance and luma sharpen (really adds that almost three-dimensional aspect to the stuff i watch [mostly anime]) - if this player could add pixel shader and the ability to use a gui, i might just leave mpc behind - great work!

[ghost]

the only thing i'm missing is pixel shaders - currently using vibrance and luma sharpen

Can you post these shaders?

[oct9]

here's links to a couple of different versions on mpc-be's site: http://dev.mpc-next.ru/index.php?topic=851.0 or you can download sweetfx shader pack on guru3d where these were originally sourced from: http://forums.guru3d.com/showthread.php?t=368880 ...........alternately there is a discussion forum on doom9 which gets very in depth (i can't run some of these on my low-end gpu): http://forum.doom9.org/showthread.php?t=157634 have fun! anyone know if there is a good tutorial on how to run mpv-player via command line on windows (i'm a real novice regarding this stuff and would love to be able to run mpv w/cuvid [if possible], change video renderer, etc)?

[ghost]

These are all written in HLSL, so it'd be necessary to port them to HLSL anyway.

Some of them look pretty simple and could just be made builtin features.

[haasn]

One of the simplest integration ideas I had could look something like this:

1. Chroma merging/upscaling and YUV->RGB conversion
2. '''User filter 1'''
3. '''User filter 2'''
4. Upscaling, etc. (including scaling pre-conversion) ...

Semantically, each filter would be a single pass that reads RGB from texture0 and dumps it into vec4 color. In the gl_rework world, this would easily integrate in a seamless way, in that the absence of a user scaler would simply have YUV->RGB conversion lead straight into scaler pre-correction, upscaling etc.

This currently lacks a way for scalers to operate on YUV, but I'm not sure if that's a huge priority since 1. there's no guarantee the input is actually YUV (it could be XYZ or RGB), 2. YUV->RGB conversion is nontrivial, so the scaler would have to operate on some arbitrary (possibly unknown) YUV space, which might even be BT.2020-CL.

This also lacks a way for scalers to operate on linear light, and I'm not sure how much of a deal that is. Possibly we could move the linear light conversion to before the scaler, but this would make scaler semantics inconsistent since it's not always done in linear light.

Finally, this lacks a way to write custom upscaling filters - although we could simulate it by allowing shaders (or the user) to request their output FBO size, so they can effectively do their own version of interpolation if they want to (before the main upscaling), the only obvious deficit is that it wouldn't benefit from sigmoidization etc.

Still, I think it's simple enough and useful enough to warrant inclusion.

[ghost]

Yes, it's simple enough as soon as we've merged the vo_opengl rewrite.

[OrdinaryMagician]

I like where this is going :3

[haasn]

I've thought about it some more, and I think two notes are in order:

1. Custom upscalers seems almost orthogonal to custom pixel shaders, in terms of the API, requirements and use case.
2. For applying shaders before vs after upscaling, the main benefit of “before upscaling” is performance, but the main benefit of “after upscaling” is quality. I think based on this, and the fact that performance shouldn't really matter much unless you're doing a very complicated filtering operation at 4K resolution or something, that only “after upscaling” is worth implementing.

I propose the following two additions to the API:

1. A custom value for ‘scale’ (resp. ‘cscale’), like scale=custom=FILE which allows you to load your own scaler from a file.
2. A suboption like ‘shader’ that lets you load multiple additional (post-upscaling) shaders, like shader=FILE1,FILE2,...

For both of these, the API would work as such: The file contains raw GLSL code that is expected to sample from sample_tex at sample_pos. It also has access to sample_size which is the size in pixels of the source texture. It is expected to return its output in the (pre-defined) vec4 color;.

Can we agree with this?

[bylee20]

@haasn

A custom value for ‘scale’ (resp. ‘cscale’), like scale=custom=FILE which allows you to load your own scaler from a file.

What about accepting window function or kernel instead of shader for custom scaler? If implementing evaluation of window function is a burden, you can just accept lua code as window function and let lua evaluate it.

A suboption like ‘shader’ that lets you load multiple additional (post-upscaling) shaders, like shader=FILE1,FILE2,...

What is preventing introducing pre-upscaling shader?

Also, if someone wants to load several shaders, he or she can just put all them into one file. I don't think you need to support multiple shader unless it's easy.

I think it's much better to support both of pre/post upscaling custom shader than multiple shader.

It is expected to return its output in the (pre-defined) vec4 color;.

Does this mean that the custom shader is a part of main function instead of separated function? If it does, custom shader cannot declare function which makes it hard to write a bit long shader.

[haasn]

What about accepting window function or kernel instead of shader for custom scaler? If implementing evaluation of window function is a burden, you can just accept lua code as window function and let lua evaluate it.

This is an API I have been thinking of redesigning, independently of the custom pixel shader support, to make it so that you can choose your kernel and window function independently. (Notably, mpv has a ton of built-in window functions, but almost nothing uses them)

Could potentially make it so you can pass a lua-based arithmetic expression, who knows. But I also don't think that it replaces custom upscaler support, since that way you still can't implement stuff that isn't based on a convolution.

What is preventing introducing pre-upscaling shader?

API. How is mpv supposed to know which scalers are supposed to run before and which are supposed to run after upscaling?

Does this mean that the custom shader is a part of main function instead of separated function? If it does, custom shader cannot declare function which makes it hard to write a bit long shader.

Yes, but mainly because implementing it this way would be easier. You raise a good point, it should probably be a custom function due to that reason alone, but this would require potentially more involved code changes.

I think it's much better to support both of pre/post upscaling custom shader than multiple shader.

How about this, then? shader=FILE and pre-shader=FILE, the latter of which runs before upscaling. Also, should the “post upscaling” shader run in linear, video-relative RGB? That would be the easiest to implement.

[ghost]

How about adding special syntax to the shader to pass information to mpv? Like the first line could be something like shader-type: scaler. mpv would parse this and remove it from the actual shader.

[haasn]

I've given a proof of concept a shot at https://github.com/haasn/mpv/commit/36dab95690e9b4d253f71759e43cb26e314a7256

One thing I noticed while playing around with it is that filtering after upscaling is undesirable for everything that doesn't involve a naive color transformation, since the signal is usually extremely low-passed by that point.

It might just be due to the very high upscaling factors I'm testing it on, though. Needs more feedback.

[haasn]

Implemented in https://github.com/haasn/mpv/commit/b0ab87976aff18e3dd5e7d0d7792f4123237708d.

I've decided to go with the “three separate options” approach, ie. you have an option for pre-shader (gets applied before upscaling), post-shader (gets applied after upscaling/subtitles, but before interpolation/CMS) and scale-shader (replaces the scaler).

The only thing I'm dissatisfied with is the fact that you still can't use custom function definitions due to limitations of the gl_sc system, so that could be reworked in the future.

[bylee20]

@haasn

Form your recent commits in your fork repo, I noticed that you're trying to expose YUV texture(YCbCr) to shader(source-shader).

Maybe this is helpful but, you'd better just give up to support non-RGB space shader if it's implementation/maintainment burden, or it makes the syntax dirty. If someone wants to do something in non-RGB space, he can do it by himself simply by multiplying conversion matrix to RGB color.

If you care about 'real original color' not assumed certain color space(BT. 709 or something), I think you'd better add a uniform matrix and its inversed one to get original color space value from RGB and vice versa.

[haasn]

The reason that exists is because some shaders MUST run before any sort of upscaling - the color is more or less irrelevant.

For example, this deband shader cannot work after conversion to RGB because it needs to deband the chroma information, which is distorted too much after the conversion. (Try it yourself, you can convert it to a pre-shader by hard-coding the extra parameters to 1)

[bylee20]

the chroma information, which is distorted too much after the conversion.

Hmm, because the color management related code and shaders in mpv are too complicated for me to understand, maybe what I'm thinking is wrong.

What is 'distortion' here? If you can define a transformation to calculate rgb, you can also define reverse transformation which calculates original input from output rgb unless the transform is non-linear, can't you? Or, does mpv perform an non-linear transform to obtain RGB?

[haasn]

No, the issue is related to subsampled video (eg. 4:2:0). The chroma planes are stored at half the luma plane's resolution.

To convert to RGB, you need a Luma and two Chroma samples for each pixel. This is why cscale exists, it upscales the chroma texture to the luma texture's size so they can be converted to RGB.

This upscaling process is the issue here: You can't reliably deblock, deband, deinterlace or do any other sort of video adjustment after it has been already distorted from its original form. (Some, more sophisticated algorithms may even rely on knowing the exact bit depth used for the input in order to process it with lots of knowledge about the raw sample values)

Or, does mpv perform an non-linear transform to obtain RGB?

Since this question is interesting in its own right: 1. the YUV->RGB conversion matrix is an affine transformation, not a linear transformation. In particular, it includes stuff like brightness/contrast, fixing the (16,235) -> (0,255) scale, hue/saturation and other factors, most of which are unknown to the shader either way. 2. Not all inputs are YUV; some are already RGB inputs, some are linear 16-bit XYZ inputs, and some are in systems like YcCbcCrc which are not linearly related to RGB.

[bylee20]

You can't reliably deblock, deband, deinterlace or do any other sort of video adjustment after it has been already distorted from its original form.

Okay. What I concerned is that it is too generalized which can make things too complicated to maintain the code. But if you want to support those filters, yeah I can't help agreeing with you.

1.the YUV->RGB conversion matrix is an affine transformation, not a linear transformation. In particular, it includes stuff like brightness/contrast, fixing the (16,235) -> (0,255) scale, hue/saturation and other factors, most of which are unknown to the shader either way.

Affine transformation is not linear, yes, you're right but you can make it linear by raising dimension by one. With alpha = 1, all affine transform in rgb space can be represented by 4x4 matrix. If you want to keep original alpha, You can save it temporarilly in other variable and restore it.

vec4 color = ...
float alpha = color.a;
color.a = 1.0;
color = affine_transform_to_rgb_4x4 * color;
color.a = alpha;

In fact, I used only one 4x4 matrix for yuv->rgb/equalizer/inversing color etc. before I migrate to opengl-cb.

2.Not all inputs are YUV; some are already RGB inputs, some are linear 16-bit XYZ inputs, and some are in systems like YcCbcCrc which are not linearly related to RGB.

This was the exact reason why I thought you should provide only rgb input. Maybe my expression wasn't enough. Unless the transform is non-linear and too complex to define its inverse transform, you can always find solution to reverse the result (except what you said in 1.).

If an algorithm depends on color space, first, you have to let the shader know given color space, and second, it still is not gauranteed to execute the shader, for instance, an source shader for YCbCr won't be executed for RGB source(such as textures obtained from vaapi/vdpau). So I thought that the usage of source shader would be very limited and it would be better and simple to let programmer write his own conversion shader in RGB space. But, what you mentioned in 1. was exactly the 'very limited cases'...

[haasn]

Right now, the semantics of source-shader is that it basically just replaces the texture() function when reading from planes, so debanding etc. works if your input is YUV or RGB - it doesn't really care about the color space at all, just the numbers themselves and their distribution.

I fully agree that you shouldn't really expect to do any color-critical transformations in the source shader, but this is also why we have a source shader, pre shader, scale shader and post shader. (Which currently all operate on different colorspaces) Different APIs for different semantic purposes.

source-shader really is most useful for things that directly counteract numerical quirks of the input encoding process, like banding or blocking.

(In fact, we could probably write a good GLSL deblocking filter)

[bylee20]

Oh btw, can you implement simple (linear) bob deinterlacer with glsl? I tried it before when I was doing everything by myself but the result was not good.

[haasn]

Just duplicating lines should be trivial to do, the more complicated part would be making anything that alternates between frames. (Maybe there should be a global frame_count uniform that just keeps counting up?)

[bylee20]

well, there's no connection between interaced fiekd and frame number, isn't it? I think a uniform representing current field needed.

Also, in that case, vo_opengl should handle interlaced frame. Or, I think it may be better to add an filter such as splitter or marker to only marks field and split into up field and down field. The real data don,t need to be copied because both field come from same frame. All these things can be done in vo but if you do this in vf, the result also can be used in other vos which can supports deinterlacing such as vo_vaapi.

[haasn]

I was semi-guessing that it wouldn't matter whether you get the odd/even frames right as long as you alternate between them, but maybe that assumption is wrong.

I've refactored the API for source-shader to make it so that extra information like this is passed along as uniforms, rather than parameters - so now we can freely add information like field_number or frame_count as uniforms without breaking existing shaders.

[haasn]

This is what a very naive deinterlacer could look like

vec4 sample(sampler2D tex, vec2 pos, vec2 size)
{
    float height = size.y / 2;
    float ybase = (floor(pos.y * height - 0.25) + 0.25) / height;
    float yoff = (frame % 2) / size.y;

    return cmul * texture(tex, vec2(pos.x, ybase + yoff));
}

It simply determines the base (by flooring to height/2) and then optionally adds an offset to it, alternating between 0 and 1.

(Note that this, by design, ignores every other frame so the result you get is actually worse than traditional bob deinterlacing. There's no way to avoid that in the VO side of things, since a shader can't somehow introduce new frames)

[bylee20]

There's no way to avoid that in the VO side of things, since a shader can't somehow introduce new frames

Of course not possible only with shader. That's why I mentioned additional 'splitter/marker' filter. Anyway, these things had better be disscussed after custom shader support merged.

Btw, do you have eta to merge those code?

[haasn]

Btw, do you have eta to merge those code?

Working on the last cleanup commits as we speak, some minor things still need changing but it should be pretty soon-ish.

[Soukyuu]

Seeing as this hasn't been merged yet, which branch/where can I check out to try it?

[ghost]

This was merged.