Improve Height blending options

TokisanGames commented 6 months ago

Description

Issues

Provide artists with more knobs for adjusting height blending
Blend more interestingly when not using height textures

How it currently works

The terrain editor is a vertex painter, not a pixel painter. The shader bilinearly blends the 4 surrounding vertices. Linear looks like a gradient. Bilinear looks like a blended square, as shown below when there are no height textures:

What varies the blend is either the height texture for use in height blending, or the application of the noise texture to the weighting. If height blending is disabled, or there is no height texture, the blend is a straight bilinear blend, aka gradient aka alpha blend.

Suggestions for blending improvements

Scale height during weighting

This scales down heights to allow for greater noise in clamping the weights by @Xtarsia :

even with height map packed, it can still not be great depending on the heightmap, uv scale etc. The shader can be tweaked a little, to give a lot more weight to the noise3 value when blending.

currently the default blend weighting is setup like this:

    weights.x = blend_weights(weights0.x * weights0.y, clamp(mat[0].alb_ht.a + noise3, 0., 1.));
    weights.y = blend_weights(weights0.x * weights1.y, clamp(mat[1].alb_ht.a + noise3, 0., 1.));
    weights.z = blend_weights(weights1.x * weights0.y, clamp(mat[2].alb_ht.a + noise3, 0., 1.));
    weights.w = blend_weights(weights1.x * weights1.y, clamp(mat[3].alb_ht.a + noise3, 0., 1.));

but if you have no height data, this yields a value of 1 in alpha giving noise no room at all.

however even with height data, due to being limited to 1m blend square, unless the height map is highly variant, and high contrast the "square" blending is going to show through in a basetexture <> basetexture case.

a very quick change is to modify the shader at this section:

    float height_bias = 0.5;
    weights.x = blend_weights(weights0.x * weights0.y, clamp(mat[0].alb_ht.a*height_bias + noise3, 0., 1.));
    weights.y = blend_weights(weights0.x * weights1.y, clamp(mat[1].alb_ht.a*height_bias + noise3, 0., 1.));
    weights.z = blend_weights(weights1.x * weights0.y, clamp(mat[2].alb_ht.a*height_bias + noise3, 0., 1.));
    weights.w = blend_weights(weights1.x * weights1.y, clamp(mat[3].alb_ht.a*height_bias + noise3, 0., 1.));

This height scale on the weighting could be applied per texture.

Contrast and Ramp

A 3D artist requested applying contrast to the height image, followed by control over a color ramp.

Contrast: new_val = (val-min) / (max-min) Ramp: user control over the min and max values of the bilerp.

https://github.com/TokisanGames/Terrain3D/assets/632766/af3471a1-62c9-447c-8175-5385ac288207

https://cdn.discordapp.com/attachments/1130291534802202735/1229003786819993640/height_map_blanding.mp4?ex=662e19f8&is=661ba4f8&hm=ea47d194938747410a91e6e5d2d6066a90ce88a3f3b605df52ee844e2a889be0&

Modify heights before weighting

In get_material when heights are looked up: height = height * scale + offset

Changing scale and offset aren't very different from each other.

Normal adjustment

@Xtarsia:

per texture height scale ( 0. to 1. range) affects the blend range

if you have gravel and cobbles setup as above, then i'd be inclined to set cobble scale to 1, and gravel scale to 0.5

I think the offset is kind of redundant for this case(especially if we can get slightly better control over painting specific blend values). However, why not apply an offset to the normals only? with 0 being current behavior, and 1 being weighted to the base texture normals even at maximum blend to the overlay? ( depending on the overlay normal height offset) with scale, and seperate height offsets of normal and albedo can have some interesting results. 1st image is extreme seperation of albedo/normal blend, 2rd image would be a more likley setup (slight push of the mud normals, sand scale at .3 with a +0.6 offset)

this would be default (0.87 blend sharpness only)

the more im fiddling with this, the less i think per-material height needs it own variable at all. Its much better to just take the time to author a decent hightmap to pack.

I think for worst case / lazy user scenario a "use noise for height blending" check box would suffice

a normal blend weight per material, ranging from -0.5 to +0.5 is subtle but effective

normal_rg = height_blend(normal_rg, albedo_ht.a + n_blend[out_mat.base], normal_rg2, albedo_ht2.a + n_blend[out_mat.over], out_mat.blend);

for negatives values the material will take on the normals of the other during blending, and vice versa, depending on the sum difference.

useing the demo grass/rock its quite nice:

with a bit of care, i can paint just the normals of the overlay now at very low blend values (same code, just manually setting per texture values in an array in the shader atm)

TokisanGames commented 6 months ago

I'm haven't fully studied or understood the normal adjustment option. I'm inclined to go with the 3D artist's request.

However, before making any decision, I want to discuss how to implement the witcher's texture blending options on pages 14-35.

GDC2014 Presentation, Gollent Marcin, Landscape Creation pdf

I'm mostly interested in how they are able to blend two textures on p28. Using the base and overlay texture in the same area, this should allow us to combine grass and rock into either:

rock with some grass in the cracks (imagine)
rock covered in grass, with the cracks showing through both (as shown)

Though all of the options and this ticket is for material wide, or per texture options, ultimately I'd like to have a paintable option where I can paint one combined material, then elsewhere, paint the other. Any thoughts on this @Xtarsia?

Xtarsia commented 6 months ago

a refrence shader i threw together to take a look at, stick it on a well subdivided plane mesh, and throw in some packed textures:


render_mode blend_mix, cull_disabled, diffuse_burley, specular_schlick_ggx;

group_uniforms test_parameters;
uniform int uv_blend : hint_range(0, 3, 1) = 0;
uniform float test_height : hint_range(0.0, 0.25, 0.001) = 0.1;
group_uniforms;
//per material values
group_uniforms material_settings;
uniform float blend_value : hint_range(0, 1, 0.01) = 0.85;
uniform float blend_sharpness : hint_range(0, 1, 0.01) = 0.9;
//normal shift pushes normal blending ahead/behind the current blend value
uniform float normal_shift : hint_range(-0.5, 0.5, 0.01) = 0.2;
//height values from the base texture below the depth threshold are ignored for the majority of a parabolic blend curve
uniform float depth_threshold : hint_range(0.0, 1.0, 0.01) = 0.2;
//height values where the dot product from the base texture, and "texel space" UP
//below the slope threshold are ignored for the majority of a parabolic blend curve
uniform float slope_threshold : hint_range(0.0, 1.0, 0.01) = 0.3;
uniform float uv_scale_0 = 2.0;
uniform float uv_scale_1 = 2.0;
group_uniforms;
//per texture values
//height scale is used to reduce the vertical coverage of the given texture
group_uniforms base_texture;
uniform sampler2D packed_albedo_0 : source_color, filter_linear_mipmap_anisotropic, repeat_enable;
uniform sampler2D packed_normal_0 : source_color, filter_linear_mipmap_anisotropic, repeat_enable;
uniform float base_height_scale : hint_range(0.0, 1.0, 0.01) = 1.0;
group_uniforms;
group_uniforms overlay_texture;
uniform sampler2D packed_albedo_1 : source_color, filter_linear_mipmap_anisotropic, repeat_enable;
uniform sampler2D packed_normal_1 : source_color, filter_linear_mipmap_anisotropic, repeat_enable;
uniform float overlay_height_scale : hint_range(0.0, 1.0, 0.01) = 1.0;
group_uniforms;

vec4 height_blend(vec4 a_value, float a_height, vec4 b_value, float b_height, float blend) {
    float ma = max(a_height + (1.0 - blend), b_height + blend) - (1.001 - blend_sharpness);
    float b1 = max(a_height + (1.0 - blend) - ma, 0.0);
    float b2 = max(b_height + blend - ma, 0.0);
    return (a_value * b1 + b_value * b2) / (b1 + b2);
}

vec3 unpack_normal(vec4 rgba) {
    vec3 n = rgba.xzy * 2.0 - vec3(1.0);
    n.z *= -1.0;
    return n;
}

float parabola( float x, float shape ) {
    return pow( 4.0*x*(1.0-x), shape );
}

//just for testing / making more sense of height
void vertex() {
    UV *= vec2(4,4);
    NORMAL = vec3(0.0,0.0,1.0);
    float slice;
    if (uv_blend == 0) { slice = blend_value;}
    else if (uv_blend == 1) { slice = UV.y*1.0-1.5;}
    else if (uv_blend == 2) { slice = UV.y*2.0-3.5;}
    else if (uv_blend == 3) { slice = UV.y*4.0-7.5;}
    slice = clamp(slice,0.,1.);

    //mat 0
    vec4 alb_0 = texture(packed_albedo_0,UV*uv_scale_0);
    vec4 nrm_0 = texture(packed_normal_0,UV*uv_scale_0);
    vec3 unm_0 = unpack_normal(nrm_0);
    float height_0 = alb_0.a*base_height_scale + 0.5-base_height_scale*0.5;

    //mat 1
    vec4 alb_1 = texture(packed_albedo_1,UV*uv_scale_1);
    float height_1 = alb_1.a*overlay_height_scale + 0.5-overlay_height_scale*0.5;

    float dot_base = dot(unm_0,vec3(0.0,0.0,1.0));
    float hmask = 0.;
    float blend_mask = smoothstep(1.0,0.5,slice);
    if (dot_base < slope_threshold || height_0 < depth_threshold) {
        hmask = mix(0.,1.,blend_mask);
    }
    float final_height = max(height_0 + (1.0 - slice),height_1 + slice);
    VERTEX.y += final_height * test_height;
}

void fragment() {
    NORMAL = vec3(0.0,0.0,1.0);
    float slice;
    if (uv_blend == 0) { slice = blend_value;}
    else if (uv_blend == 1) { slice = UV.y*1.0-1.5;}
    else if (uv_blend == 2) { slice = UV.y*2.0-3.5;}
    else if (uv_blend == 3) { slice = UV.y*4.0-7.5;}
    slice = clamp(slice,0.,1.);

    //mat 0
    vec4 alb_0 = texture(packed_albedo_0,UV*uv_scale_0);
    vec4 nrm_0 = texture(packed_normal_0,UV*uv_scale_0);
    float height_0 = alb_0.a*base_height_scale + 0.5-base_height_scale*0.5;

    //mat 1
    vec4 alb_1 = texture(packed_albedo_1,UV*uv_scale_1);
    vec4 nrm_1 = texture(packed_normal_1,UV*uv_scale_1);
    float height_1 = alb_1.a*overlay_height_scale + 0.5-overlay_height_scale*0.5;

    //blend out normal Shift
    float n_shift = mix(0.0,normal_shift*2.,parabola(slice,0.5));

    //depth and slope threshold
    vec3 unm_0 = unpack_normal(nrm_0);
    float dot_base = 1.0-dot(unm_0,vec3(0.0,0.0,1.0));
    if (slope_threshold > dot_base || height_0 < depth_threshold) {
        height_0 = mix(height_0,height_1+1.0,parabola(slice,0.125));
    }

    vec3 albedo = height_blend(alb_0,height_0,alb_1,height_1,slice).rgb;
    vec4 norm_rough = height_blend(nrm_0,height_0,nrm_1,height_1+n_shift,slice);

    ALBEDO = albedo;
    NORMAL_MAP = norm_rough.rgb;
    NORMAL_MAP_DEPTH = 1.5;
    ROUGHNESS = norm_rough.a;

}

full range blend: overlay on top, but not in the cracks: overlay filling the cracks

Xtarsia commented 6 months ago

I suspect it might be reasonable to have a bunch of values, have them all per texture (to establish decent defaults)

and then, with some remaining bits from the control map, instead of painting slope/uv scale/rotation etc, use 7 or even 8 bits, to paint a "material" index. (as @TokisanGames has been hinting at)

this would then allow upto 256 different materials, each with a completley unique look.

the arrays could contain: slope/depth/base_tint/overlay_tint/base_uv/overlay_uv/normal_shift/base_roughness/overlay_roughness/kitchen_sink etc..

TokisanGames commented 6 months ago

Great work. Did you use The Witcher team's algorithm?

How do we get 256 materials?

32 A
32 A on B
32 B on A =96

Or are you counting A at 0 degrees, A at 90 degrees as two different materials?

Xtarsia commented 6 months ago

should expand on what exactly is needed to be stored, and where?

painting a "material" applys a full set of data simultaneously via a "material brush" anywhere where

off the top of my head,

example "paintable material": (control map) material index = sandy red rocks (control map) base mat = rock (control map) over mat = sand (control map) blend value = 1.0

MOST IMPORTANT: (material array) material blend value = 0.56

This is a seperate blend value, the control map blend lerps all ranged settings from their texture defaults to the material indexed values as the (control map)blend value goes from 0 - 255) This should allow some form of blending between a Material, and no Material. Blending between 2 material would only occur through the bilinear blend however.

texture defaults means that every texture has a full default set of these parameters as well, except for the material specific blend value above

THE REST: blending (material array) normal_shift(to base) -0.5 (almost entirely ignores the sand normals except where sand height value is near max) (material array) roughness_shift = 0.5 (almost entirely ignores the rock roughness except where sand height value is near max) (material array) blend sharpness = 0.92 (material array) height_blend_filter = 0.0 (filters overlay by base height value, ie set to 0.5 to only cover the top half of the height) (material array) slope_blend_filter = 0.0 (filters overlay by base normal slope, ie set to 0.5 to only cover mostly flat area)

texturing (material array) base color tint = red/brown (material array) over color tint = white (material array) over uv = 0.1 (default as inherited when first creating the material) (material array) base uv = 0.75 (make the rocks bigger for this material) (material array) over_rotation_noise = 1.0 (material array) over_rotation = 0.0 (material array) base_rotation_noise = 1.0 (material array) base_rotation = 1.0

world filter (material array) world_height_falloff_begin = 50.0 (above 50m in world space, begin fading out blend value) (material array) world_height_falloff_end = 60.0 (at 60m in world space, blend value is faded to 0) (material array) slope_min = 1.0 (flat, in world space, including the base surface normals) (material array) slope_max = 0.5 (45 degrees, including the base surface normals)

after this is painted, the shader reads the material index from control map, and applies all material array settings for that index to the blending process.

it would comepletley be possible to go and directly paint other base/overlay textures etc after the fact, whilst the material is still set. in this case we could consider "material index" to really be a control bit for "select from this list of pre-defined blending parameters"

as for the witcher, i just looked at the pictures and thought "yep, can do that" :P

Xtarsia commented 6 months ago

as for how do we get 200+ materials:

this is all useing the same base/over mat, and in 1 case the same texture twice! Just different blend options: sand <> gravel

magma rocks <> obsidian

muddy rippled sand

gravel with slate chunks

chunky red clay <> wavey sand

Xtarsia commented 6 months ago

it may even be possible in the future to ship Terrain3D with 4 default CC0 textures loaded, and around 30 pre-configured materials

edit: I do think this should be about the limit of complexity, and that most use cases would just be setting per-texture defaults (which could have slightly different behaviours depend on if the texture is set as base or overlay)

TokisanGames commented 2 months ago

weights.x = blend_weights(weights0.x * weights0.y, clamp(mat[0].alb_ht.a + noise3, 0., 1.));

In #466 @NectoT suggested mix(mat[0].alb_ht.a, noise3, 0.5) in lieu of the clamp to work with textures without heights. This is similar to the height bias given above.

With the luminance to height generation in #471, there's no reason to not use height textures, if one wants that look. No performance gain or vram savings. However, we want to provide more height blending options, which could facilitate textures without heights.

TokisanGames commented 1 month ago

We'll get some more height blending improvements into 0.9.4.

There's a shader from Xtarsia I need to test still. height_blending_shader.txt https://discord.com/channels/691957978680786944/1185492572127383654/1268618276184002641

And blending reference