No stencil buffer write with direct3d11/12

[lampysprites]

added hxcpp results in the end

Discussed in https://github.com/Kode/Kha/discussions/1415

^{Originally posted by **lampysprites** April 20, 2022} Hello! I'm trying to create a mask for drawing using stencil buffer. Works in html5 as intended but something seems to be missing in other cases. In addition to the example below, tried drawing to an explicitly created Depth24Stencil8 image before the buffer. Appreciate the help. os: windows10 graphics: geforce gtx 560 kha: main @ [f4fd02a5fd9a4bbc8133d985d1646a16498c44de](https://github.com/Kode/Kha/commit/f4fd02a5fd9a4bbc8133d985d1646a16498c44de) ### How it looks 

Code

package;

import kha.Assets;
import kha.Framebuffer;
import kha.graphics4.PipelineState;
import kha.graphics4.VertexData;
import kha.graphics4.VertexStructure;
import kha.math.FastMatrix3;
import kha.Shaders;
import kha.System;

using kha.graphics2.GraphicsExtension;

class Main {
    static var pipeMask:PipelineState;
    static var pipeOver:PipelineState;

    static function render(fb:Framebuffer) {
        var t = kha.System.time,
            g = fb.g2;

        g.begin();

        g.pipeline = pipeMask;
        g.color = 0xffffffff;
        g.fillCircle(400, 300, 200);

        g.flush();

        g.pipeline = pipeOver;
        g.color = 0x88ff00ff;
        g.pushTransformation(FastMatrix3.translation(400, 300).multmat(FastMatrix3.rotation(t)).multmat(FastMatrix3.translation(150, 0)).multmat(FastMatrix3.rotation(-2 * t)));
        g.fillRect(-150, -150, 150, 150);
        g.popTransformation();

        g.end();
    }

    public static function main() {
        System.start({title: "Kha", width: 800, height: 600}, function (_) {
            Assets.loadEverything(function () {
                var structure = new VertexStructure();
                structure.add("vertexPosition", VertexData.Float32_3X);
                structure.add("vertexColor", VertexData.UInt8_4X_Normalized);

                pipeMask = new PipelineState();
                pipeMask.inputLayout = [structure];
                pipeMask.vertexShader = Shaders.painter_colored_vert;
                pipeMask.fragmentShader = Shaders.painter_colored_frag;

                pipeMask.stencilBothPass = Replace;
                pipeMask.stencilMode = Always;
                // pipeMask.colorWriteMask = false;
                pipeMask.stencilReferenceValue = Static(1);

                pipeMask.compile();

                pipeOver = new PipelineState();
                pipeOver.inputLayout = [structure];
                pipeOver.vertexShader = Shaders.painter_colored_vert;
                pipeOver.fragmentShader = Shaders.painter_colored_frag;

                pipeOver.stencilMode = Equal;
                pipeOver.stencilReferenceValue = Static(1);

                pipeOver.compile();

                // Avoid passing update/render directly, so replacing them via code injection works
                System.notifyOnFrames(function (framebuffers) { render(framebuffers[0]); });
            });
        });
    }
}

Seems that it is indeed directx thing. After trying different options, here's the results

• direct3d11 doesn't show the box at all
  • hen clearing with 1's, and circle being 0s, looks same as krom above. kinda as if only reading works?
• opengl - all works fine
• direct3d12 - same as krom, no masking
• direct3d9 - empty window even with default pipeline

using cl.exe == 19.29.30133 x64 in case that matters