Use a stencil buffer for region clipping

[Lupo511]

Adds a stencil buffer and uses it for region clipping. This is just a pretty basic and straightforward implementation that only has rectangle clipping, but it could actually be expanded to support clipping to any shape.

[xerpi]

That looks pretty good, although I'd add something like: vita2d_disable_clipping() so the draw to the stencil buffer doesn't have to be done on each frame. Also try to follow the coding style and place { on the same line as the if.

[Lupo511]

Done!

[xerpi]

It's better, but I would do it a bit differently:

vita2d_enable_clipping: clipping_enabled = 1 and SCE_GXM_STENCIL_FUNC_EQUAL
vita2d_disable_clipping: clipping_enabled = 0 and SCE_GXM_STENCIL_FUNC_ALWAYS

vita2d_set_clip_rectangle: saves clipping coordinates to global variable and clip_rectangle_dirty = 1

at vita2d_start_drawing:
if (clipping_enabled && clip_rectangle_dirty) {
    perform a pass to the stencil;
    clip_rectangle_dirty = 0
}

I can do this modifications myself if you want.

[Lupo511]

Wouldn't that stop vita2d_set_clip_rectangle from working while in the scene?

[xerpi]

Oh I see, with my way vita2d_set_clip_rectangle is only updated at the start of each frame, and that's bad if you want to change the clipping region.

So what about this:

vita2d_set_clip_rectangle()
{
    if  (drawing) {
        perform a pass to the stencil;
    } else {
        save clip coordinates;
        clip_rectangle_dirty = 1;
    }
}

vita2d_start_drawing()
{
    ...

    if (clipping_enabled && clip_rectangle_dirty) {
        perform a pass to the stencil;
        clip_rectangle_dirty = 0
    }
}

[Lupo511]

I thought that sceGxmBeginScene cleared the depth and stencil buffers, that's why I was resetting it at every frame, but that might actually be wrong.

[xerpi]

I think the stencil buffer is untouched, so I guess we can use the same stencil without touching it across scenes.

[Lupo511]

That's perfectly fine then.

[Lupo511]

By the way saving clip coordinates should probably be done both when drawing and when not, since they're used in the get_clip_rectangle function.

[xerpi]

That's right :)

[Lupo511]

It's ok if you want to add this yourself, it's a quick change so I'm fine either way.

[xerpi]

Finish it yourself if you don't mind :)

[Lupo511]

Sure! :)

[Lupo511]

Are you sure that beginning the scene doesn't clear the stencil buffer? I'm getting different results.

[xerpi]

I think we have to disable writing to the stencil when performing the screen clear.

[Lupo511]

You mean in vita2d_clear_screen?

[xerpi]

Yeah, it draws a rectangle to clear the screen, so we have to disable stencil buffer writes and fetches while clearing.

[Lupo511]

But the way it's now shouldn't write to the stencil buffer already.

[Lupo511]

I don't know I keep getting results that seem to indicate that the stencil buffer is cleared (or something is clearing it).

[xerpi]

If the stencil buffer is enabled, then calling vita2d_clear_screen will also use the stencil, right?

[Lupo511]

Yeah but it shouldn't modify it, like if you were using draw_rectangle or circle or anything, since after setting the active rectangle on the stencil within set_clip_rectangle, the stencil function is set back to keep the stencil value.

[xerpi]

Right. Then just do:

vita2d_start_drawing()
{
    ...

    if (clipping_enabled) {
        perform a pass to the stencil;
    }
}

and we can forget about clip_rectangle_dirty.

[Lupo511]

Ok! So wait what needs to be changed now?

[xerpi]

Maybe nothing, I'll take a deeper look and test myself later.

[Lupo511]

Ok 👍

[xerpi]

Merged #41