Depth testing for Triangle primitives

[KABoissonneault]

I use im3d to draw some 3d overlays over a 3d scene in an entirely separate render pass, after the scene.

Sometimes, logically, the overlays are meant to traverse some entities in the scene, but visually, the overlays are always rendered on top.

In the example image above, the blue quad (drawn using DrawQuadFilled) is supposed to traverse the cube, but it's hard to see without moving the camera around to understand where the quad is located.

I was wondering if anyone has experimented with depth testing for im3d, at least for the triangle primitives. I was able to get a ID3D11DepthStencilView* from my 3d scene, and provide it to my D3D11 im3d implementation, and bind it using OMSetRenderTargets, while binding a depth-stencil state with depth testing enabled.

However, I could not get results that made sense for the im3d primitives. The overlays would have parts that randomly appeared and disappeared as the camera moved in closer or farther.

I kind of expected point and line primitives to have issues, but even the blue quad in the screenshot above would have nonsensical results.

[john-chapman]

I'm not sure that I exactly understand what you want to achieve, here are some thoughts:

If you're trying to have Im3d primitives depth-tested against your existing scene, this should work. A common approach for this is to directly sample your scene depth texture in the Im3d pixel shader and modify the alpha so that pixels behind the scene depth are more transparent, which looks like this:

However from your example, I guess that you're trying to have Im3d primitives depth-tested against each other, which isn't going to work perfectly because they're alpha blended. As you've seen, parts of the blue quad will be discarded by the depth test as your camera moves around, which is probably not what you want.

The classical solution to this problem is painter's algorithm, which involves sorting primitives by depth and rendering them back-to-front. Im3d actually supports this. Sorting is enabled like draw state with PushEnableSorting()/PopEnableSorting(). Note that the sorting is only per primitive so it's still not going to solve your problem completely - in order to have the blue quad intersect the box correctly you'll need per-pixel sorting or some implementation of OIT.

[KABoissonneault]

Oh, no, I want depth testing against the scene - the cube is rendered in a pass before im3d, using another (retained) graphics library. The white wireframe and blue vertices don't matter here.

I hadn't thought of sampling the depth buffer in the pixel shader. Is there a reason for taking this approach, rather than just using the depth testing offered by graphics APIs?

On the other hand, since writing this issue, I realized my graphics library had reversed depths by default, which is why I had weird results mixing them with im3d. I've disabled this, and while it's not perfect yet, I think I'll just have to run it through RenderDoc and figure out why im3d and my other libraries have mismatching depth values

[john-chapman]

Ah ok I understand.

Is there a reason for taking this approach, rather than just using the depth testing offered by graphics APIs?

I just suggested it as it's a common approach for overlays if you wan to have the overlay remain slightly visible when it intersects the scene depth.

Enabling depth testing while drawing Im3d should work fine, as long as you use the same view/projection transformations as when rendering the scene. Of course you'll still see some z-fighting if you have Im3d triangles exactly coplanar with your scene.

[KABoissonneault]

Alright, so my issues are probably related to mismatching view-projection matrices. I think I'm good otherwise.

I'm probably going to fork this to add 2 extra VertexList vectors (ie: m_vertexData) that set a new "depth testing enabled" flag in the DrawLists, so I can enable and disable depth testing in the rendering implementation depending on the elements. Would you be interested in such a pull request?