search

codecat / nanovg_d3d

A Direct3D 9 and 11 port of nanovg, an OpenGL 2D vector drawing library for UI and visualizations.
zlib License
7 stars 3 forks source link

nanovg_d3d9: Stencil for path filling is bugged #1

Open pabloko opened 3 years ago

pabloko commented 3 years ago

Fills rendered with D3D9nvg__fill seems to have wrong stencil operations as the CCW part is always kept.

https://i.imgur.com/WFpynHs.png

Maybe the two setencil mode is needed here?

Regards

codecat commented 2 years ago

Seems like this might be a problem on dx11 as well:

image

pabloko commented 2 years ago

Forwarding this issue, i had some time to look at nvg_d3d9 today and learn about the pipeline used in it.

So, starting by replicating the issue with the least possible interferences, AA disabled:

    nvgBeginPath(vg);
    nvgFillColor(vg, nvgRGBA(255,0,0,255));
    nvgMoveTo(vg, 100, 100);
    nvgLineTo(vg, 100, 0);
    nvgLineTo(vg, 200, 100);
    nvgLineTo(vg, 300, 0);
    nvgLineTo(vg, 400, 100);
    nvgLineTo(vg, 500, 0);
    nvgLineTo(vg, 500, 100);
    nvgClosePath(vg);
    nvgFill(vg);

image

Rendered on a web canvas looks like this: image

And then dump the cached paths image

So, shapes are rendered to the stencil buffer then masked off a bounding quad, in an attempt to tesselate the shape, unfortunatelly the combination of trianglefan primitive and the stencil modes dont seem play nice in d3d9, things i tried so far: using two side stencil mode, reorder verts from counterclockwise to clockwise, and many stencil flags... in short, the tesselation is broken.

But i want something usable for a project i will need soon, so i replaced the tesselation and added earcut.hpp tesselator library, wich generates indices from the geometry vertices, then i added indexed primitive drawing support for fill operations, resulting on a correct render:

image

And heres the demo

image

ill work in a PR when i have some time for it and polished the index buffer allocator but my solution is suboptimal and im still looking at fixing the tesselation

codecat commented 2 years ago

Thanks for looking into this! Wouldn't you want triangle strips instead of triangle fans?

pabloko commented 2 years ago

Thanks for looking into this! Wouldn't you want triangle strips instead of triangle fans?

It wont make much sense, if you look at the dumped data of the shape, they're just sequential points, wich only allows trianglefan to be used, looking into implementation details, the GL implementation uses glStencilOpSeparate with GL_FRONT and GL_BACK stencil buffers, wich means the front-facing and back-facing triangles, in other hand, d3d9 has a concept of two-sided stencil, wich on paper should be the same, but it seems specifically respecting the winding order of the paths, so it seems theres no matching api on d3d9 for this gl feature, or im not able to replicate...

one way to check this would be draw the stencil, then draw it again with inverted set of points and using an EQUAL stencil comparison to just keep the pixels matching in both stencils, while it should produce the correct shape, theres an extra primitive rendering wich is better to avoid.

other approach may change the geometry inverting some points to make the two-side stencil mode to work.

in my case, i did not like both of those solutions, so ive used a tesselator library that just generated indices for indexed primitive rendering, this allows to use the same vertex data that is already on the pipeline, but render them as TRIANGLELIST with the indices generated by the tesselator.

This way i can remove the convex fill, as the tesselattor supports it, and all the consecutve shapes can be done in a single call, as every triangle is independent.

I still have to fix the AA fringes, but i dont know if i will do it as i have enabled multisample for this call and the result is good

So, no AA, no convex and indices allocator are a bit broken but this is shaping correctly at least: https://pastebin.com/uCvz2KAD

image