(apparent) Shader pixelshader parser/compiler bug.

[gee-ell]

Operating System Info

Windows 10

Other OS

No response

OBS Studio Version

27.1.3

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/Me2NHJSdFZ7a3TRo

OBS Studio Crash Log URL

No response

Expected Behavior

pixel shader should output the specified sampled pixel, regardless of what is computed/sampled in the function.

Current Behavior

While writing a simple custom shader for the obs-multisource-effect plugin, I seem to have hit a shader parser/compiler bug. I'm not familiar with OBS' shader system, but I'm a long-time D3D shader effect programmer so this is clearly wrong:

Using obs-multisource-effect's 'add.effect' example - this just adds together two texture inputs, and works correctly:

uniform float4x4 ViewProj;
uniform texture2d src0;
uniform texture2d src1;

sampler_state def_sampler {
    Filter   = Linear;
    AddressU = Clamp;
    AddressV = Clamp;
};

struct VertInOut {
    float4 pos : POSITION;
    float2 uv  : TEXCOORD0;
};

VertInOut VSDefault(VertInOut vert_in)
{
    VertInOut vert_out;
    vert_out.pos = mul(float4(vert_in.pos.xyz, 1.0), ViewProj);
    vert_out.uv  = vert_in.uv;
    return vert_out;
}

float4 PSDrawBare(VertInOut vert_in) : TARGET
{
    float4 rgba0 = src0.Sample(def_sampler, vert_in.uv);
    float4 rgba1 = src1.Sample(def_sampler, vert_in.uv);
    return rgba0 + rgba1;
}

technique Draw
{
    pass
    {
        vertex_shader = VSDefault(vert_in);
        pixel_shader  = PSDrawBare(vert_in);
    }
}

To demo the bug: when outputting only the src0 texture like this, it works:

float4 PSDrawBare(VertInOut vert_in) : TARGET
{
    float4 rgba0 = src0.Sample(def_sampler, vert_in.uv);
    float4 rgba1 = src1.Sample(def_sampler, vert_in.uv);
    return rgba0;
}

However, trying to output the src1 texture gives the src0 texture instead (this is wrong):

float4 PSDrawBare(VertInOut vert_in) : TARGET
{
    float4 rgba0 = src0.Sample(def_sampler, vert_in.uv);
    float4 rgba1 = src1.Sample(def_sampler, vert_in.uv);
    return rgba1;
}

It only works correctly if you remove/comment out the src0 sampler line (should not be required):

float4 PSDrawBare(VertInOut vert_in) : TARGET
{
//  float4 rgba0 = src0.Sample(def_sampler, vert_in.uv);
    float4 rgba1 = src1.Sample(def_sampler, vert_in.uv);
    return rgba1;
}

I hit this bug (without a workaround) in a similar simple shader, which just takes the rgb from src0, and the alpha from src1 (breaks the same way, only one texture is used for both inputs). But as adding the two rgba sources does work correctly, this has to be a bug in the shader parser and/or compiler?

Steps to Reproduce

You may be able to repro simply, but my steps were:

1. Install obs-multisource-effect (https://obsproject.com/forum/resources/multi-source-effect.1412/)
2. duplicate its add.effect to [custom name].effect
3. make the indicated changes to output only src0 or src1 (while still sampling both)
4. apply the plugin, set two video or image sources and use the custom shader file.

result: outputting src1 results in src0 instead (as detailed).

Anything else we should know?

again note that in the example, if the sampler line for src0 is removed, then src1 is output correctly.

[gee-ell]

this is my custom shader that also breaks in the same way (the same texture is incorrectly used for both inputs):

float4 PSDrawBare(VertInOut vert_in) : TARGET
{
    float3 video = src0.Sample(def_sampler, vert_in.uv).rgb;
    float   mask = src1.Sample(def_sampler, vert_in.uv).a;
    return float4(video, 1.0 - mask);
}

[norihiro]

I've added a comment to the original issue. https://github.com/norihiro/obs-multisource-effect/issues/5#issuecomment-987764868 I don't know what you actually did. Please close this issue if a workaround that I mentioned in the comment works.

[gee-ell]

@norihiro, this is not about the shader caching issue I mention in the linked thread, but about the pixel shader outputting the wrong sampled texture, depending on the actions it performs. this seems to be a bug in Obs' shader parser or compiler.

[norihiro]

There is another bug in the plugin to render alpha channel. I fixed the bug in the release 0.1.5. Does this issue persist with the updated plugin?

[gee-ell]

yes, the Obs parser still seems to 'optimise out' one of the texture reads incorrectly in the examples given.

[gee-ell]

latest OBS code pull seems to fix the issue.

[gee-ell]

sorry, my 'src0 = rgb, src1 = alpha' example now works.

But the 'only outputting src1' example still outputs src0 incorrectly.

[norihiro]

Could you attach your shader file (not paste the contents but drag&drop the file) so that I can test it with exactly same file.

[gee-ell]

shaders.zip

[norihiro]

I confirmed the issue on Windows10 running on Virtualbox. The issue is not confirmed on Linux. OBS version: 2021-12-08-b92e941b4-27.1.3-win64 (the latest master branch, built on Github Actions)

[FiniteSingularity]

I've run into this issue with the latest OBS 29. The first example:

float4 PSDrawBare(VertInOut vert_in) : TARGET
{
    float4 rgba0 = src0.Sample(def_sampler, vert_in.uv);
    float4 rgba1 = src1.Sample(def_sampler, vert_in.uv);
    return rgba1;
}

is definitely still an issue- src0 is the texture that is returned, rather than src1 which should be returned.

Further, I have greyscale mask that I'm trying to apply as the mask for another source (r, g, b are all the mask value, and a is 1.0):

float4 PSDrawBare(VertInOut vert_in) : TARGET
{
    float4 image = src0.Sample(def_sampler, vert_in.uv);
    float4 mask = src1.Sample(def_sampler, vert_in.uv);
    return float4(image.rgb, mask.r); // incorrectly returns float4(image.rgb, image.r)
}

Uses the red channel from the original image, rather than the red channel from the mask. HOWEVER, if I add an active alpha channel to my mask, and sample from mask.a rather than mask.r, it works, e.g.-

float4 PSDrawBare(VertInOut vert_in) : TARGET
{
    float4 image = src0.Sample(def_sampler, vert_in.uv);
    float4 mask = src1.Sample(def_sampler, vert_in.uv);
    return float4(image.rgb, mask.a); // correctly returns float4(image.rgb, mask.a)
}