Arvtesh / UnityFx.Outline

Screen-space outlines for Unity3d.
MIT License
1.26k stars 90 forks source link

RenderingLayerMask not set correctly? #53

Open Wyvernius opened 2 years ago

Wyvernius commented 2 years ago

Hi,

When setting Outline Filter Settings to Rendering Layer Mask its not applied correctly. I have currently 2 outlines setup, one on layer 2(yellow) and other on layer 3(blue) as all objects are assigned to layer 1 by default.

Outline set to layer 3 gets rendered everywhere (all objects that are set to 1) When i set layer 3 to layer 5 it also gets rendered on all objects with Layer 1 and Layer 3.

So it seems to me that all even and odd numbers in the layer have the same effect.

The whole world is outlined and all those renderers are set to their default (RenderLayer1). Capture

Wyvernius commented 2 years ago

I think i found the issue. In OutlineSettingsWithLayerMask.cs the OutlineRenderingLayerMask was just returning an uint, without bit-shifting.

public uint OutlineRenderingLayerMask
        {
            get
            {
                if (_filterMode == OutlineFilterMode.UseLayerMask)
                {
                    return uint.MaxValue;
                }

                if (_filterMode == OutlineFilterMode.UseRenderingLayerMask)
                {
                    return (uint)(1 << (int)_renderingLayerMask);
                }

                return 0;
            }
        }

Now i can correctly assign Layermask in renderer by: renderer.renderingLayerMask = 1 << 2

To get the outline to render that i've selected for layer 2.

Arvtesh commented 2 years ago

Hey @Wyvernius, thanks for the report. I'll look into this. I believe a better solution would be to have an editor script that displays a selection popup of all available rendering layers (like the one in MeshRenderer).

Wyvernius commented 2 years ago

That is indeed a better solution. Was looking at it for a bit, but just needed it to be fixed.