exeldro / obs-shaderfilter

OBS Studio filter for applying an arbitrary shader to a source.
GNU General Public License v2.0
377 stars 39 forks source link

Improper blending when alpha is zero but RGB values are not (0,0,0) #30

Closed FiniteSingularity closed 8 months ago

FiniteSingularity commented 8 months ago

Shaderfilter rendering was recently change to use obs_source_process_filter_begin_with_color_space when rendering the output rather than obs_source_process_filter_begin ( https://github.com/exeldro/obs-shaderfilter/blob/618008b688f6a4004eda35164a063ae235cf8a21/obs-shaderfilter.c#L1164 ). This was likely done to fix some color blending issues when alpha is neither zero nor one.

However, it is causing a problem (both here, and in my plugins) where for alpha values of zero, the RGB values must be zero as well, otherwise the output alpha channel is incorrect. For example, given a very simple shader:

float4 mainImage(VertData v_in) : TARGET
{
    float alpha = distance(v_in.uv, float2(0.5, 0.5)) < 0.2 ? 1.0 : 0.0;
    return float4(1.0, 0.0, 0.0, alpha);
}

This should render a red ellipse in the center of the source, with transparent alpha outside of the circle. In earlier versions of shaderfilter this is exactly what happens: image

However in the new version, the pixels outside of the ellipse are rendered with a very high (nearly 1.0) alpha: image

If we alter the shader slightly to ensure that all pixels outside of the ellipse have their RGB values set to (0.0, 0.0, 0.0) things render as they should: image

I suspect the reason is that both the alpha, and the magnitude of the RGB values are being used for the output channel. Perhaps it has something to do with the blend functions set here? https://github.com/exeldro/obs-shaderfilter/blob/618008b688f6a4004eda35164a063ae235cf8a21/obs-shaderfilter.c#L1173