Array Shader - not blending well in some cases (sharp square)

[RonanZe]

Describe the bug Squares are appearing when painting with a high opacity value (or staying to much time at the same place).

A screenshot to illustrate:

I'm maybe wrong but this my analysis after some tests in the code:

The shader seems not to work well when one "weight" of the array is set to his max value (1.0) --> ie w(1, 0, 0, 1) Should the shader handle better this case or does the paint_indexed_splat() method should always keep a minimum value to blend 2 textures/layers? --> w(0.9, 0.1, 0, 1)

Environment

• OS: Win8.1
• Graphics card RTX2080
• Godot version 3.2.2beta1
• Plugin version, master 1.3
• Renderer used: GLES3

[RonanZe]

I just try to force the weight to stay at a minimum value but it's not solving the problem. I have to learn more how the shader works 😅

[Zylann]

Maximum values should work fine. How did you obtain this result? Which texture indexes did you paint? Is it the plugin's brush or procedural generation?

As explained in the doc, it is known that this shader is recent and tricky to get right, but I could get it to work ok after a few iterations (see screenshots in the doc). So I don't know what to do with this issue other than acknowledge that yes, using it is hard, the brush is hacky and isn't as nice as I want. I don't have an answer to that yet^^"

---

From my findings so far, the main rule is, some textures may not be able to blend, and interpolation can play nasty tricks. So I had to come up with a compromise with the brush algorithm, which is a lot more complicated than regular painting...

Important: in the following text, I'm listing weights as 3 components, but in reality, that texture only has 2 components, R and G. The third one is inferred as 1 - r - g, because the sum of all 3 must be 1. So if you generate that procedurally, don't forget to compress B.

The indexes can be set to anything, but there can be only 3 active at once, and its not possible to blend between two pixels if the textures use the same component. I could not figure out a way to nicely assign components dynamically, so each texture gets assigned a fixed component of the index map, R, G or B, automatically. So 1 goes to R, 2 goes to G, 3 goes to B, 4 goes to R again etc.

For example, let's say you want to paint the two following textures (1 and 5, which have non-shared components), and have them blend:

weight(1.0, 0.0, 0.0) index(1, 2, 3) <-- will use R
weight(0.0, 1.0, 0.0) index(4, 5, 6) <-- will use G

This is how it's supposed to look like, in the blending area:

weight(1.0, 0.0, 0.0) index(1, 2, 3)
       0.8, 0.2, 0.0        1, 5, 3
       0.5, 0.5, 0.0        1, 5, 3
       0.2, 0.8, 0.0        1, 5, 3
weight(0.0, 1.0, 0.0) index(4, 5, 6)

As you can see, the weight of texture 1 will decrease, while the weight of 5 increases. 3 and 6 don't matter, it could have been something that was painted there before. But the original pixel does not have 5, and the final pixel doesnt necessarily had 1. This can cause an artifact in the very ends of the blend, so I also attempted to make sure that if a pixel fully has the same texture, I set all components to the same index, like this:

weight(1.0, 0.0, 0.0) index(1, 1, 1)
       0.8, 0.2, 0.0        1, 5, 3
       0.5, 0.5, 0.0        1, 5, 3
       0.2, 0.8, 0.0        1, 5, 3
weight(0.0, 1.0, 0.0) index(5, 5, 5)

It had positive results, but again it's far from perfect when lots of textures start to overlap.

When painting more over this, the brush basically "transfers" weights to the component you are painting towards. But if you use a texture that shares the same component as one you painted already, there isn't a way to blend. The index will change from 1 to 4, but the weight is already 1.0, so... you get a hard transition. So to avoid this, the brush will first attempt to transfer the weight of that compoment to the others, and once it's zero, it swaps it to the new one. It still means the blend will not be direct since other textures are going to appear, but at least it makes the transition smooth, and can be covered by a transition texture (also described in the doc).

For example, if you have a map covered with 1:

weight(1.0, 0.0, 0.0) index(1, 2, 3)

And you want to paint with 4:

weight(1.0, 0.0, 0.0) index(4, 2, 3)

This is what the brush does:

weight(1.0, 0.0, 0.0) index(1, 2, 3)
       0.8, 0.2, 0.0        1, 2, 3
       0.5, 0.5, 0.0        1, 2, 3
       0.2, 0.8, 0.0        1, 2, 3
       0.0, 1.0, 0.0        4, 2, 3 <-- hard switch of R here,
       0.2, 0.8, 0.0        4, 2, 3     but should not be visible since weight is 0
       0.5, 0.5, 0.0        4, 2, 3
       0.8, 0.2, 0.0        4, 2, 3
weight(1.0, 0.0, 0.0) index(4, 2, 3)

But I'm still being played by texture filtering unfortunately, so there doesn't seem to completely avoid occasional edges.

I'm aware editing textures with this shader is harder, and figuring out how to paint it was annoying. I just don't know how to proceed to an upgrade of that. I have some ideas in progress but there are problems left so can't apply them.

[RonanZe]

Thanx for your more in depth overview. It's more easy to understand why a brush stroke don't always works has expected. But I still need to think a bit more to figure out the whole picture :student:

When reading the doc, I thought that you where using some kind of static index like that:

index(1, 2, 3)
      4, 5, 6
      7, 8, 9

And when your painting, you only should care to avoid having blending between 1 and 4 for example. But It's seems more tricky than that.

You explain it but just to be sure; in the current implementation, there are cases when you have mixed combination like this (2, 4, 3) ? Or you never change the order for 1/4/7... --> always in R 2/5/8... --> always in G 3/6/9... --> always in B like that: (4, 2, 3)?

Is it because the way the shader works? Where does this constraint come from?

If I understand correctly, you need to avoid hardswitch in the weight_map+index_map if you want the shader to interpolate fine? Is it the vec3 get_depth_blended_weights(vec3 splat, vec3 bumps) in the fragment shader that define the interpolation? with the smoothstep? I'm still need to understand a little bit better how interpolation works at shader level...

What to you mean by "being played by texture filtering unfortunately" ?

---

To answer you first question:

I'm using the plugin's brush. Basically, I'm painting with the texture 7 over the default one. I guess the 0. It's hard to understand exactly why those gliches appears. Sometimes, it's work ok and when I downsize the brush, there are more visible:

[Zylann]

You explain it but just to be sure; in the current implementation, there are cases when you have mixed combination like this (2, 4, 3) ? Or you never change the order for 1/4/7... --> always in R 2/5/8... --> always in G 3/6/9... --> always in B like that: (4, 2, 3)?

Is it because the way the shader works? Where does this constraint come from?

Each texture index is assigned a fixed component, so 1 will always be in R, so is 4. 5 will always be in G, etc. It's not really required by the shader itself, it's just an attempt I made at constraining the situation because dynamically assigning indexes made the artifacts appear in unpredictable ways. With this constraint, I thought it would make it deterministic, so easier to workaround when painting. But again, there is more to it, which remains to be solved.

What to you mean by "being played by texture filtering unfortunately" ?

What the brush attempts to do is to transfer weights and set indices when it deems it safe. The problem is, the brush only does that in relation to pixels. In reality, even if two pixels are valid, neighbors matter too, because the GPU will interpolate in between, and that can cause unwanted artifacts in the middle. If you had these two pixels for example:

index(1, 2, 3) weight(1.0, 0.0, 0.0)
index(4, 5, 1) weight(0.0, 0.0, 1.0)

They are valid, and technically represent the same result. But if the GPU interpolates that, this will appear in the middle:

index(1, 2, 3) weight(1.0, 0.0, 0.0)
      1, 2, 3         0.5, 0.0, 0.5
index(4, 5, 1) weight(0.0, 0.0, 1.0)

That makes texture 3 appear, which isn't wanted. That's one reason why I decided each texture can only appear in the same component AND turned off filtering on the index map. Unfortunately, this remains the same blocker that prevents me from using other ideas I have. At first I thought, if a pixel makes a neighbor become wrong, I could "fix" the neighbor too. But then that "fix" could make a neighbor of the neighbor wrong too, so I need to fix it too. And then a neighbor of the neighbor of the neighbor could be wrong, etc... not manageable^^

---

Regarding your problem, I think I see what you mean. I get that too, however I'm not sure yet why that happens. Might be the result of another workaround I did^^" The brush has soft edges, but if you keep painting the same spot over, the result will accumulate to hard-edged, making this happen? And it would happen more on small brushes because their falloff is quicker, due to them being smaller. Seems like the best workaround is to first paint the area, and paint on the edges with a softer brush to smooth them out.

[RonanZe]

Yes. You describe well the main issue. But sometime, and I don't know why, it seems more glitchy than other. I have to redo the tests because I wondering if it's not related to the way I'm selecting brush at start. But I may be wrong and it's only when I'm changing size. I don't know If I can gather more infos on that..

---

For the interpolation problem, is it possible to write in the shader a custom interpolation method?

Because one of the main aspect of the problem seems to be coming from the fact that the interpolation in gpu is based by channel and is not compatible with the current index system.

[Zylann]

Update, turns out https://github.com/Zylann/godot_heightmap_plugin/issues/172 could have had an effect on this (thought not entirely sure if it would have done anything, depends if Godot messed up as sRGB by default or not). It doesn't mean sharp corners will now go away though.

I'm already working on another kind of shader, still using texture arrays, but giving better results, with up to 16 textures.

[RonanZe]

Great! After our discussion, I'm also thinking about a shader that using one more "layer" made of pure RGB colors has coordinate. To avoid to directly blend the weight texture. Index+weight are more like data attached to the vertex/poles.

The idea is to have per triangle full Red on first vertex, full Blue on the 2de and full Green on the 3th. With that, you can have the coordinates of the fragment in the triangle and remap the data given by the poles.

If think it can help avoid hardswitch, simplify the problem because you only deal with one triangle at a time (and not trying to have a weight+index map that work in every side on different triangles).

I'm still not sure if it best to use vertex color or a map of fading RGB texture with UVs. And I did'nt take a look of the way you create the triangles for the Terrain...

[Zylann]

I don't quite understand what you describe. I don't even deal with triangles here, remember there is LOD going on, geometry can change.

The shader I'm working on is a mix between the array one and classic4, except it uses 4 RGBA splatmaps, which is where comes the 16 possible textures. But intsead of sampling 32 textures (16 albedos and 16 normalmaps) it filters the 4 highest weights and interpolates only 8, the same way as classic4. There is still some sort of limitation if too many textures blend at the same place but it sounds like it will be far easier to work with. Once this shader works, there is another possible upgrade to bring it more textures, but I'll probably see about that much later.

[RonanZe]

Seem like a good idea for a more straightforward shader!

---

Yes, I know the LOD can be a problem. But I think that it can works with UVs to.

It probably difficult to explain without a visual but it's not that far from how a simple Gouraud shader is working.

If you know in the fragment function how far you are from a vertex (between 0.0 and 1.0) and what textures are blending form that vertex, you can interpolate the 3 textures discribes in that vertex --> index(.., .., ..) weight(.., .., ..)

Do that for the 2 other vertex of the triangle and mix them together.

Here is a more visual attempt to explain the idea. The gradients in the triangle are linear so not totally correct and you have think that there are texture. The black gradiant is just to how it's blending:

[Zylann]

Your example works at the corners, but don't forget the fragment shader runs for every single pixel of the triangle. Sampling one texture is actually 2 lookups: albedo and normal. So that's 6 samples when just at a vertex. Which brings us to 18 samples just to get it from a pixel in the middle, because you got potentially 3 textures from the 3 corners to interpolate. All that, if we know which triangle we are in in the fragment shader. It looks like you want to re-implement interpolation manually. I considered that at some point but I ended up dropping it due to a far higher number of required samples compared to what I'm trying now (which can blend 1 more texture in 6 less samples). Do you think you'd be able to test your theory by modifying one of the existing shaders?

[RonanZe]

Yes, the idea come from our conversation and how to avoid hardswitch by redoing the interpolation. I also read that an other well know super splat shader + tool was working with vertex color...

But after thinking on how to use UV instread of vertex color, I'm seeing it more the RBG gradients like a mask.

I'm testing a partern right now. But I will not have a lot of time to convert it to a shader before next week. And I'm also not really fast writing shader :smile:

This is my first try:

The green channel looks like this:

But I think the others one are not correct:

Should look all like diamond.. or not

Maybe it's to much computation, of maybe I miss something. I'll see after more experimentations

[RonanZe]

I have a working result!

array-v2.zip

To summarize: This shader is a derivative of @Zylann Array shader --> it use an index_map and weight_map to list the textures and blend amount. The difference here is that the interpolation is decouple from the weight_map.

The interpolation is compute in the shader with modulate function on UVs to know where the fragment is in a cell: vec2 fragment_coords = vec2(mod(1.0 - UV.x, 1.0 / terrain_size), mod(1.0 - UV.y, 1.0 / terrain_size)) * terrain_size;

The downside, is that there is a lot more additions, multiplications and 4 times of texture fetching.

I only add the the ALBEDO atm. No Bump, normal, rougthness or depth blending.

It should be easy to add but I still need to create an nice array texture to test those parts.

To use this shader, you need to turn off the filtering of the "splat_weight.png" I have disable it in the hterrain_data.gd by changing texture_flags = Texture.FLAG_FILTER to texture_flags = 0 Don't know if it's the good way to do it but it seems to works.

@Zylann How are you doing yours speed tests? is it in the "Profiler" ? Can you try it in the same conditions has the multiplats?

[Zylann]

@RonanZe I do my speed tests with a MeshInstance on which I use hterrain_mesher.gd to generate one chunk of 256x256, then I setup a shader material on it with textures, and place a camera right above such that the plane covers the whole screen. No lights, no environment. Then I turn off vsync and measure frames_drawn per second in fullscreen. But to be comparable it needs to test all shaders on the same setup. Basically I make sure to reproduce a typical polycount while having the minimum things running in the scene, so timings should be as representative as possible of the shader's execution time.

I just benchmarked it, it scores 1.52 ms. That's a better result than multisplat if you only need albedo and no depth blending (and no triplanar), however if you need the rest it will be higher.

[RonanZe]

Thanks for the bench.

I tought using custom interpolation and not the one coming from textures was a lot slower. Seems to be equivalent.

I'll add other parts and see what is the final cost.

[RonanZe]

Can't find frames_drawn. Is it in the Profiler? Monitors?

I have this method Engine.get_frames_drawn() but don't seems to be precise enough

[Zylann]

Here is the script I use: test_shader_performance.zip

You may have to recreate the scene and make representative textures though.

[RonanZe]

@Zylann Thanx for the script. Works great. Doing depth blending is not has easy has I tought. But you already guest that :smile:

To do that, the shader need to recreate all the possibilities. It can go up to 16 weights/alpha maps and that's a lot. Maybe I can reduce this number but there is still the need to reordre the indexes of the texture and that can a lot of time if it's not done in a smart way.

Maybe there is clever tricks using bit mask on floats instead of vec4 but I don't have enough knowledge in this area.

Your multiplats stay the better option. I wonder how it's done elsewhere. I can't find a lot of informations about this subject