Idea: Overwrite layered entity / block entity rendering - banners, horses, etc

[coderbot16]

Currently, to blend gradients and different textures together on layered content like horses and banners, Minecraft repeatedly draws the same model with different textures / blending modes. This might mean that rendering a complex horse or banner could require upwards of 16 draw calls.

This leads to a number of issues, including performance concerns, as well as not working well with Iris batched entity rendering. In addition, it is not compatible with deferred-rendering shaderpacks, which are not able to properly handle translucency in many cases.

A smarter approach would be to place all of these different layers into a 2D texture array, and then have a shader that is capable of sampling from this array in a loop as needed. It's likely that some sort of auxiliary UBO or similar would be necessary to pass information about the ordering of layers.

Then, existing vanilla / Iris shaders could be patched to read from this array instead of sampling the main texture. This would alleviate challenges with having to pass horses & banners through Iris batched entity rendering, it would allow deferred-rendering shaderpacks to handle horses & banners properly without any special work needed for translucency, and finally it could improve performance as well.

This would have some mod compatibility & implementation complexity concerns. In addition, it might be more fit for a mod like Sodium that is oriented towards optimization & replacing vanilla rendering.

[coderbot16]

Note: This would require some way to arbitrarily redirect any shader texture() call to one of our functions, but only in the case of accessing a few specified samplers. It's unclear how exactly this could be implemented using glsl-transformer at this time.

[douira]

Similarly to the case of texture() redirection for a specific method of anisotropic filtering, this kind of redirection is very difficult since without additional information or constraints it becomes Turing complete as shaders may pass samplers around dependent on complex computation. There are however a few approaches that could make this possible:

1. Heuristic approach: Even without dataflow analysis we can sometimes determine which sampler is being used if it's using the globally declared samplers without storing it as a parameter or variable
2. Limited data flow analysis: This is very difficult, but could provide a better approximation. For this, simple situations in which samplers are passed as values through variables or parameters could be solved. The implementation of this is very complex.
3. Hinting: Shaders provide information about which sampler they are using (or if the texture() call should be redirected). They could do this by either calling a different function like texture_special() which Iris then patches to the correct call. It would also be possible that the shader inserts a #custom trace_sampler <sampler name> or something similar before the function call that the patcher finds and uses to redirect the following function call.
4. There may be further approaches for tracing the origin of samplers within complex data and control flows that rely on special restrictions of GLSL. (I don't know of any, but the discussion about tracing in the AF discussion may contain some?)

[coderbot16]

I wonder if it would be possible to have a "sidecar" variable following the sampler variable around. Basically, transforming the code such that there is a corresponding sidecar variable that can store extra data for every sampler. So that would look something like this:

uniform sampler2D albedo;
uniform sampler2D other;

in vec2 texcoord;

vec4 myTexture2D(sampler2D sampler, vec2 coord) {
    return texture(sampler, coord);
}

void main() {
    sampler2D myAlbedo = albedo;
    // ...
    vec4 alb = myTexture2D(myAlbedo, texcoord);
    vec4 smth = myTexture2D(other, texcoord);
    // ...
    gl_FragColor = alb * smth;
}

being transformed into something like this:

uniform sampler2D albedo;
uniform int albedo_sidecar;
uniform sampler2D other;
uniform int other_sidecar;

in vec2 texcoord;

vec4 iris_redirectedTexture2D(sampler2D sampler, int sampler_sidecar, vec2 coord) {
    // ...
}

vec4 myTexture2D(sampler2D sampler, int sampler_sidecar, vec2 coord) {
    return iris_redirectedTexture2D(sampler, sampler_sidecar, coord);
}

void main() {
    sampler2D myAlbedo = albedo;
    int myAlbedo_sidecar = albedo_sidecar;
    // ...
    vec4 alb = myTexture2D(myAlbedo, myAlbedo_sidecar, texcoord);
    vec4 smth = myTexture2D(other, other_sidecar, texcoord);
    // ...
    gl_FragColor = alb * smth;
}

This would potentially be fully robust, and allow us to make the shader compiler do the dataflow analysis for us instead of us doing a fragile reimplementation of it.

[douira]

Interesting approach! Basically, you're suggesting we don't actually do a static anylsis but rather pseudo-attach some data to the sampler that identifies it which we can then use to check if it's the right one at runtime. (thus removing the turing-completeness of completely semantically analyzing the program ahead of time) We can probably even do it without uniforms but constants that the transformer knows about and can insert into the redirection logic. It would need to produce a sidecar variable and assignments of it for every assignment that acts on variables of the sampler type. Still challenging, but much less so than trying to do static data flow (heuristic) analysis.

[coderbot16]

Yep! This is similar to some previous ideas but without the limitation of needing to do something weird & unsupported like putting a sampler2D object into a compound data type like a struct. You're right that uniforms are not needed here (I wasn't sure how to best express the sidecar concept). But hopefully the concept is sound, since replacing texture sampling behavior unlocks some interesting improvements like AF as you mentioned, as well as this feature.

[douira]

We'll have to investigate if all the weird edge-cases of assignment expressions can be covered with this. Thankfully there are no math operation on samplers that we have to worry about. I can imagine this would increase the size of the code somewhat since every variable (uniform, local variable, parameter) of the sampler type will need a sidecar. Maybe making this optional would be a good idea, if we find that it reduces performance or compilation stability/speed.

[razorchase49]

Is this idea still being considered, and do you propose it will fix the problem of horse markings appearing over horse armor?

[douira]

I believe the sidecar concept is viable but it requires glsl-transformer to offer a bunch more semantic analysis, which it currently doesn't, to actually work. I can't give an estimate for when it will be implemented.

[douira]

As a partial solution, if there is enough interest I think the hinting concept, in which shader packs tell Iris about which sampler they are using, might also work. But it would only work with shader packs that specifically make use of the feature and I'm not sure if there are other constraints that make this approach infeasible.