Outlines diverge at certain camera angles with SetOutlineDepth::Flat

[mxgrey]

I've stumbled across a strange bug where certain camera angles will cause the rendered outline to jump off the object and diverge into the background.

I've created a gist the reproduces the issue. It does not happen with SetOutlineDepth::Real. Changing the model_origin for SetOutlineDepth::Flat does not seem to change the effect, positively or negatively.

Here's are screenshots from the gist which illustrates the problem:

Okay

Diverging

The gist itself is animated to make the issue more clear.

I'm pretty clueless about what could cause this. I would guess that vertices which have fallen out of the camera view are somehow getting reflected inside the GPU, possibly off to infinity, but I have no idea why that would happen. Especially since it seems to happen for Flat but not Real.

[komadori]

This bug is caused by my performing the conversion to NDC space inside the vertex shader. I implemented it that way in order to eliminate variance in the fragment depths (z * w / w != z due to rounding) that was causing Z fighting between entities and their stencils. Unfortunately, as you've discovered, this doesn't work because it breaks the clipping of triangles which are behind the camera.

I think a small constant depth offset in the stencil should provide an alternative solution to the Z fighting so that the vertex shader can use the correct coordinate space. I'm working on it.

[komadori]

Fixed by 0df3735906445a653547b5639479fd8c2ec5b71e.

I'll make a new 0.3.x release soon.

[komadori]

Released 0.3.4.

[mxgrey]

Thanks for addressing this so quickly!

I thought I should mention, it seems like SetOutlineDepth::Real no longer works with release 0.3.4. That doesn't affect me personally because I only want to use Flat, but I'm sure you didn't intend that outcome.

[komadori]

Oops! Thanks for letting me know. I need to add an example that uses it. I've released 0.3.5 with a fix.