Open QbieShay opened 1 year ago
It would be super useful and great if someone picks up this work code-wise. Ideally we'd be able to test WIP builds and give feedback in an iterative manner with the community. I do not have the capacity to implement this.
Thank you for your work on this proposal!
Is there any reason the StandardMaterial3D stencil options only seem to allow specifying a color, and not an entire outline/xray material? Is it for ease of implementation? I don't think this API will find use if color is all you can specify. Technically an outline/xray material could be realized by godot automatically tweaking the stencil settings of the material sub-resource when it is used for outline/xray. If the inspector allows for it, disable or hide any options which are forced to a fixed value by the parent material.
Note that stencil stuff is all way above my head, so I probably wouldn't be able to benefit from even the mid-level API, let alone anything low-level.
Is there any reason the StandardMaterial3D stencil options only seem to allow specifying a color, and not an entire outline/xray material?
The idea is that if you want a custom one you can make a shader yourself, but if you want something quick it's just a matter of a few clicks. Note that it auto-adds a second pass to the standard material.
Is it possible to support "half". So instead of an integer 0-255, it's a 16 bit integer? Not sure about the internals of allowing 32bit integers.
@fire No, OpenGL and Vulkan only support an 8-bit mask/reference value
I'll be starting work on implementing this shortly!
No idea how long it might take. I'll be trying to implement the proposal as written, I don't think there will be any problems with it.
Thank you @apples !
@QbieShay I'm wondering about the OUTLINE and XRAY modes adding a next_pass
.
What specifically should happen for this? If the regular next_pass
property is used, it will appear in editor, and may lead to confusion for users, and being able to edit that material might lead to problems. And what if the user has a next pass of their own?
Wondering what your thoughts are on this.
Hmm. I'd say for now leave it to the side, we'll have a chat about it in rocket chat. I have thought about it a bit, and I don't quite know.
I think maybe the following would work:
If user does not have a next_pass material Create and set a next_pass material with the right stencil settings
f user has a next_pass material Emit a warning in an editor toast (or do a proper popup) that says next_pass already exists, this effect requires the next pass to work.
@clayjohn I was thinking perhaps it can be together in that mesh menu where you can generate collission and outline mesh instead?
A design question regarding the reference value and comparison operators:
With this design, the stencil reference value is exposed, but not the associated compare/write masks. But in both OpenGL and Vulkan, there are compare/write masks which are used to control which bits of the stencil values are considered.
To be useful without being able to specify the masks, the assumption must be that the masks will be equal to the reference value. For example, with a reference value of 4
, both the compare mask and write mask should also be 4
.
This works fine for simple "layer" based effects, but it brings into question what the comparison operators are for. The operators EQUAL
, NOT_EQUAL
, and ALWAYS
make perfect sense to include, but when you think about the operators LESS
or GREATER
, those make less sense. GREATER
especially wouldn't make sense, as it would be equivalent to NEVER
.
So my suggestion is: Either we must allow the two masks to be specified in addition to the reference value, or we should remove the LESS
and GREATER
operators from the list.
Also, was there any thought to integrate https://github.com/godotengine/godot/pull/73527 into this? Sometimes depth functions are needed for certain stencil effects, e.g. Wind Waker style lights, which need to use depth function GREATER
.
Edit for clarity: The linked PR https://github.com/godotengine/godot/pull/78542 does include my depth function changes, but these are not part of this design as-written.
@apples IMO the masks should always be 255. Specifying masks is way too confusing for most people and only useful in very limited scenarios. This proposal intentionally ignores masks.
masks should always be 255
But then how would we support multiple different effects using different bits of the stencil value? If you have an outline effect using ref 1
, and a light effect using ref 2
, with a mask of 255
they would interfere with each other.
Remember, the goal is not to support every single possible effect with stencils. The goal is to provide something to the user that is easy to use and useful. If we expose everything, stencils will be too confusing for anyone to actually use. Plus it contributes to a huge amount of bloat in the material (which hurts usability even more).
Many popular effects that require stencil won't work with a StandardMaterial anyway as they require fine grained control over culling, render order, blending, and depth testing. In order to make them work, we would have to not only expose every feature of stenciling, but massively overhaul how the StandardMaterial works. The cumulative effect would be turning it into a tool that is slightly more flexible for very advanced users, but impossible to manage for regular users.
We want all users to be able to benefit from stencil, which is why this proposal is aimed at covering 99% of use-cases. For advanced users we have plans to expose a new way of authoring materials that will provide access to all features from the graphics API. See also https://docs.godotengine.org/en/latest/contributing/development/best_practices_for_engine_contributors.html#cater-to-common-use-cases-leave-the-door-open-for-the-rare-ones
@clayjohn Maybe there's a miscommunication here. I understand all that. I'm trying to simplify the design, not expand it.
I think using a mask of 255 is a bad idea, because it prevents multiple stencil effects from working together. As in my example, you couldn't have a character with an outline also be lit by a wind waker style light, since they would be overwriting each other's stencil buffer values.
I agree that exposing the masks to the user is a bad idea. That's why my alternative suggestion is to always use the ref value as the masks (and remove the LESS
and GREATER
operators since they become confusing at that point). This makes the ref value behave much more like a layer selection.
Isn't the power of the masks that you can use different read and write masks? If we don't support it then I'm not sure it's better than a single ref value, since ref can be always cleared afterwards (that windwaker thing can have a last stencil reset pass)
@apples Since I didn't see any further conversation on ref vs mask, I think in general if you believe there's things with mask that can't be achieved with ref only and not vice-versa, it's fine for me to add (please @clayjohn confirm), but then it should be given a name that makes is possible without breaking compat to add ref later (so, giving it a different name than ref, just mask
or layers
. i think i prefer layers
)
@QbieShay Just to clarify: I don't think we need to expose the mask to the users. I just think that using 255 as the masks internally will result in different stencil effects colliding with each other, and confuse users.
I think it would be simpler for users if we set the mask to be equal to ref, so that different stencil effects don't interfere with each other in the first place. This way there's no need for an extra "clear" pass (which is impossible to do in certain scenarios). Though, this does limit users to the 8 individual bits.
For advanced users, we can consider exposing the masks to the stencil_mode
in the shaders only (not in the material properties), but I think it should be left out for now.
I think that it would be quite confusing then to use greater and lesser, if the ref is also the mask (for example 1 wouldn't be lesser than 2). If i understood correctly. If we use masks like this, i think it should be 100% clear that it's only 8 layers so that people know how to use it
any updates on this?
any updates on this?
To my knowledge, nobody is currently working on implementing this. It's kind of depending on https://github.com/godotengine/godot-proposals/issues/7916 either way, which isn't planned before 4.4 at the earliest.
Describe the project you are working on
Godot engine
Describe the problem or limitation you are having in your project
After gathering feedback in #3373 After testing with https://github.com/godotengine/godot/pull/78542 After a significant amount of hours spent by clay and myself to wrap our head around this
Considering we don't want to expose the full complexity of stencil outside of lower level API (more on this in the future) Considering we don't want to 100% copy any other engine in their implementation (expect tutorials from Unity to not apply)
Read along for Godot's stencil proposal ^^ We've tried to think of the use cases brought up in #3373 and they all seem possible with this API.
This will be an iterative process and more features will come little by little. This is huge work. There's a lot of things to consider. Please be patient
Describe the feature / enhancement and how it helps to overcome the problem or limitation
Supersedes https://github.com/godotengine/godot-proposals/issues/3373
Stencil operations are needed for a wide range of 3D FX. They can also be very useful to optimize a lot of other 3D FX.
Currently, Godot allocates a stencil buffer when allocating the 3D renderbuffers (combined depth + stencil), but does not make use of it.
Stencil operations as exposed by the OpenGL and Vulkan standards are cumbersome, confusing, and exposes a lot of meaningless combinations of similar-sounding settings. We want to implement something that can be used by most users and not just by users with a background in advanced tech art or 3D graphics APIs.
At the same time, there is a huge demand to have some control over stencil operations to do more than just the most common effects (masking and xray).
Masking works like the depth buffer. In other words, at some point in time you render to the stencil buffer and place a value. At another time (or at the same time) you read from the stencil buffer, compare the value and reject based on a set condition (equals, not equals, greater than, etc.). Masking allows users to implement outlines, complex masks, and performance optimizations.
Xray allows users to add overlays that persist through scene geometry. I.e. show an enemy through a wall etc. Xray requires writing a stencil value at one point (i.e. when you first draw the enemy), but only when the depth test passes. In a later pass the "xray" effect is drawn and rejects any pixels that match the xray stencil value.
The important difference between the two is that Masking writes regardless of the depth test.
Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams
Shader API
Similar to
render_mode
, add the keywordstencil_mode
Mode (enum):
WRITE_DEPTH_FAIL
// stencil value is only written if the depth test failsREAD_ONLY
// stencil value is never written, but it is tested // must pass stencil and depth // depth testing can be disabled in the depth settings. Stencil testing can be disabled by not using stencilREAD_WRITE
// stencil value is written if ref passes stencil test and depth testWRITE_ONLY
// stencil value is written if it passes depth testDISABLED
Compare mode (enum):
COMPARE_LESS
COMPARE_EQUAL
COMPARE_LESS_OR_EQUAL
COMPARE_GREATER
COMPARE_NOT_EQUAL
COMPARE_GREATER_OR_EQUAL
Ref (an integer 0-255)
These could be implemented as rendering modes or with a new token
stencil_mode
i.e.:stencil_mode read_only, compare_less, 1;
Shader
The stencil value should be exposed in the fragment shader by reading from the stencil buffer with a
STENCIL_ID
keyword.Pipeline details
There's a lot of work needed to make this fully usable not only in term of exposing the relevant keywords to shaders, but to also make sure that those operations are placed in a sensible manner in the rendering pipeline. For example
WRITE_DEPTH_FAIL
cannot be run at the depth prepass. Similarly, stencil operations need to run in a consistent manner and to run either in depth prepass or in subsequent passes, but ideally not both, which causes effects to misbehave.Considerations on how depth sorting buckets are managed for opaque objects must be taken into consideration, possibly supporting appropriate ordering of subsequent draw passes specified via the "next pass API" which currently doesn't seem to sort objects reliably, causing flickering and completely breaking stencil operation.
Tentative step-by-step
We want users to be able to implement stencil outlines and stencil Xray effects without understanding how stencils work (the same way that users can implement transparent effects without understanding all the depth settings). Accordingly, we propose to expose a high level API in the StandardMaterial3D that makes these much easier.
stencil_mode (enum):
OUTLINE. When selected, exposes:
color
thickness
ID
Internally this would:
set mode to
WRITE_ONLY
set ref to
ID
Add a
next_pass
with a basic opaque material set tounshaded
and withalbedo_color
set to the specified color and with the grow property set tothickness
and with stencil mode set toREAD_ONLY
, compare mode set toNOT_EQUAL
and ref set toID
XRAY. When selected, exposes:
color
ID
Internally this would:
Set mode to
WRITE_DEPTH_FAIL
set ref to
ID
Add a
next_pass
with a basic transparent material set tounshaded
and withalbedo_color
set to the specified color and with stencil mode set toREAD_ONLY
, compare mode set toNOT_EQUAL
and ref set toID
CUSTOM. When selected, exposes:
read_write_mode
,compare_mode
(only visible whenread_write_mode
includes reading), andref
In parallel stencil explorations for 2D can be made. I have no idea how to do that or even if it's possible. I have exhausted my energy on this so proposals for 2D stencil are welcome. Feel free to experiment with the original PR I opened to then make a proposal!
Alternatives
To the people inevitably unsatisfied with this solution: we know. We hear you. We have decided to not compromise entirely on usability for this work, but it doesn't mean that it will be forever this way. There's the intention to extend the rendering API and to eventually offer low level access to the renderer and to the whole stencil API via the lower level interface. This is out of scope for this initial work. This proposal is a weighted compromise between:
Conclusion
Thank you for reading until here if you have, and thanks to everyone that participated to this discussion and provided their input. Special thanks to @apples whose initial work made it possible at all to finally reach a consensus on how we want this API to look like.
Thank you all <3
If this enhancement will not be used often, can it be worked around with a few lines of script?
No
Is there a reason why this should be core and not an add-on in the asset library?
No