Stencil is broken in 12.0-development

[er2off]

In development branch of future 12.0 version function love.graphics.setStencilMode does not drawing anything using syntax love.graphics.setStencilMode('keep', 'greater', 0, 0, 0, function)

Changing keep with other StencilActions and read/write masks still gives nothing.

Linux amd64 with OpenGL renderer Love2D commit d586d1847446f5212d5f7e9efb94e50fcfba7d77 SDL2 2.0.22

[slime73]

setStencilMode doesn't have a parameter which expects a function.

See https://github.com/love2d/love/commit/6dbb7fa5de6c04491676ac09cede3d49fcea0168

[er2off]

As I understand, need to write to buffer. But I don't really understand how?

[EngineerSmith]

This is an example I wrote for someone else on discord for the new flow of stencils in love12

local lg = love.graphics
-- love 11.4, previous stencil flow
love.draw = function()
  lg.stencil(function() -- define the stencil
      lg.rectangle("fill", 50,50,100,100)
    end, "replace", 1) 
  lg.setStencilTest("greater", 0) -- allow rendering on pixels which have a stencil value greater than 0 e.g. where our rectangle is
  lg.rectangle("fill", 0,0,150,150)
  lg.setStencilTest() -- cleanup
end
-- love 12, new stencil flow
love.draw = function()
  lg.setColorMask(false) -- stop drawing colours, as we only want to draw to the stencil
  lg.setStencilMode("replace", "always", 1) -- start stencil
  lg.rectangle("fill", 50,50,100,100)
  lg.setStencilMode("keep", "greater", 0) -- set stencil test
  lg.setColorMask(true) -- allow drawing colours
  lg.rectangle("fill", 0,0,150,150)
  lg.setStencilMode() -- cleanup
end

It's important to note that this new workflow lets you now draw to the stencil and the colour backbuffer at the same time. This can decrease draw calls where you wanted to. e.g.

local lg = love.graphics
local image = lg.newImage(...)

love.draw = function()
  lg.setStencilMode("replace", "always", 1)
  lg.draw(image, 50,50) -- draw our image with stencil mode on + draw colours
  lg.setStencilMode("keep", "greater", 0)
  lg.rectangle("fill", 0,0,150,150) -- draw a "background" around the image
  lg.setStencilMode()
end

Previously you would of had to draw the image twice; or a rectangle for the stencil for the image.

[er2off]

Thanks! I made backward compatibility for old API.

if love.getVersion() >= 12 then
  function love.graphics.stencil(fn)
    love.graphics.setColorMask(false)
    love.graphics.setStencilTest('replace', 'always', 1)
    fn()
    love.graphics.setColorMask(true)
  end
  function love.graphics.setStencilTest(cmp, val)
    if cmp
    then love.graphics.setStencilMode('keep', cmp, val)
    else love.graphics.setStencilMode()
    end
  end
end

[er2off]

No, problem was not solved. I don't test this on PC because I was on mobile. This code does not works.

    love.graphics.setColorMask(false)
    love.graphics.setStencilMode('replace', 'always', 1)
    --love.graphics.stencil(function()
      love.graphics.rectangle('fill', x, cy + oy, w, h, r)
    --end)
    love.graphics.setStencilMode('keep', 'greater', 0)
    love.graphics.setColorMask(true)

    -- some draw calls

    love.graphics.setStencilMode()

[a13X-B]

Would this better be left for forum or discord support since it isn't the issue with love framework per se?

[er2off]

Idk how this bug appears.

If anyone wants to see what's wrong

https://user-images.githubusercontent.com/46158549/178098408-d0b43d72-d58d-4592-a176-700b908a0ed4.mp4

[slime73]

There were a couple different issues here:

• The OpenGL implementation of love.graphics.setStencilMode didn't properly implement the 'keep' stencil action (now fixed in 258d233).
• Your replacement for love 11's stencil() doesn't behave the same: aside from the missing arguments, love 11's stencil() clears the stencil buffer every time it's called (unless the 'keepvalues' parameter is set to true when calling it), and it resets the stencil testing state after executing the function parameter.

Clearing the stencil buffer before using it will be expensive especially if it's used multiple times per frame. Instead of emulating love 11's approach there, I recommend using different stencil values for different things in a frame.