Extending ShadowMaterial

[dan-edington]

I'd like to extend the three.js ShadowMaterial so I can apply shadows to my custom shader. When I try to do this my shader code gets ignored and the default ShadowMaterial is rendered (ie. the fragments in shadow render as black, the rest of the mesh is transparent).

Looking at the source code for ShadowMaterial, I can see it explicitly sets the gl_FragColor, whereas the other materials use outgoingLight as their output:

gl_FragColor = vec4( color, opacity * ( 1.0 - getShadowMask() ) );

Am I approaching this wrong or is this not possible with the library currently?

[FarazzShaikh]

That is correct. Since ShadowMaterial is a nonstandared material type, and does not use the conventional pattern for its output, CSM does not currently support it by default.

For now you can use the patchMap prop to manually define an override for your use case like so: https://codesandbox.io/p/devbox/hardcore-mendel-8qrc2x?file=%2Fsrc%2FApp.tsx%3A21%2C10

The goal with CSM is to support all ThreeJS in-built materials out of the box so I will mark this as a feature request and work on it in the next update.

[dan-edington]

Thanks for the reply! I wasn't able to access your codesandbox for some reason but the solution I ended up going with for my use case was to extend MeshLambertMaterial and inject the shadowmask_pars_fragment shaderchunk so I could access the getShadowMask() function.

[FarazzShaikh]

Sorry the box was set to private. Should be fixed now. But yes thats essentially the solution:

    <CSM
        baseMaterial={ShadowMaterial} //
        fragmentShader={
          /* glsl */ `
          void main() {
            float alpha = opacity * ( 1.0 - getShadowMask() );
            csm_DiffuseColor = vec4(1.0, 0.0, 1.0, alpha);
          }
        `
        }
        patchMap={{
          "*": {
            "gl_FragColor = vec4( color, opacity * ( 1.0 - getShadowMask() ) );": /* glsl */ `
              gl_FragColor = csm_DiffuseColor;
            `,
          },
        }}
      />

[CosyStudios]

Can I assume this issue is the case with extending MeshDepthMaterial too?

Guessing id need to patch the following last line if viable.. `float fragCoordZ = 0.5 * vHighPrecisionZW[0] / vHighPrecisionZW[1] + 0.5;

#if DEPTH_PACKING == 3200

    gl_FragColor = vec4( vec3( 1.0 - fragCoordZ ), opacity );

#elif DEPTH_PACKING == 3201

    gl_FragColor = packDepthToRGBA( fragCoordZ );

#endif`

I'm trying to patch a shader (juniorsounds depthkit.js) which uses a large geometry and a video texture to set all non relevant pixels as discarded but the depth image, seems to still be the original geometry & in my particular case, this causes a big rectangle to overlay on top of what should be outlines in an outline pass...

[CosyStudios]

... working in vanilla three , not r3f if thats relevant at all

[CosyStudios]

Shouts out for the exceptional work thus far though , really glad to find this repo.

[FarazzShaikh]

If it works in Vanilla but not in R3F then it must have to do with your setup in react (Memoization and such). If you can post a minimal reproduction I can take a look

also remember that some pathways in ThreeJS materials require certain properties to be set on the material instance. So for example in MeshDepthMaterial, the depthPacking property value determines which pathway gets executed

[CosyStudios]

Beg your pardon, I meant I'm working in vanilla three, not r3f - as in i cant get the depth material to work in vanilla three, Im not using react at all.

I'll see if i can create a sandbox in due course but might be tricky as involves video files..

My rough attempt so far is essentially a second CustomShaderMaterial with the same vertex/frag shader as the actual mesh material and a patchmap doing: const depthPatchMap = { "*": { // The keyword you will assign to in your custom shader "float fragCoordZ = 0.5 * vHighPrecisionZW[0] / vHighPrecisionZW[1] + 0.5;": " float fragCoordZ = 0.5 * csm_DiffuseColor.z / csm_DiffuseColor.w + 0.5; " }, }

• which comes befores the pathway to choose depth packing which uses fragCoordZ, so i thought that would cover that.

Confused about the usage of " // The keyword you will assign to in your custom shader" - im just using * all of the time as Im unclear as to where / how to use the keyword activation

[CosyStudios]

For further clarity - the material is setup as follows =>

this.frontDepthMaterial = new CustomShaderMaterial({ uniforms: UniformsUtils.clone(Uniforms), baseMaterial: MeshDepthMaterial, vertexShader: crossDkVert, fragmentShader: crossDkFrag, patchMap: depthPatchMap, transparent: true, cacheKey: () => { return this.props.reference + '-depth-front' }, }) ..... this.mesh.customDepthMaterial = this.frontDepthMaterial;

[FarazzShaikh]

I see. As for the comment -

// The keyword you will assign to in your custom shader

CSM will only inject your patch map if the keyword you specify in place of * is present in your shader. This is useful if you are building generic patchmaps for multiple material types. However, using * bypasses this requirment and injects your patch map regardless.

Most if not all third party patch maps would use * so you are on the right path

I am not too sure what the intended result is here. Perhaps an running example would help. If you are trying to set the depth based on a video texture here is what the patchMap would look like:

const crossDkFrag = `
    uniform sampler2D tDepthVideoTex;
    varying vec2 vUv;

    void main() {
        vec4 depth = texture2D(tDepthVideoTex, vUv);
        vec4 csm_CustomDepth = depth;
    }
`;

const depthPatchMap = {
  csm_CustomDepth: {
    "gl_FragColor = packDepthToRGBA( fragCoordZ );": `
            gl_FragColor = csm_CustomDepth;
        `,
  },
};

And then you would simply set the RGBADepthPacking to THREE.RGBADepthPacking (as RGBADepthPacking = 3201)

Note my useage of csm_CustomDepth in place of *. This would mean that if you removed vec4 csm_CustomDepth = depth; from your frag shader, the patch map will not be injected thus not breaking your shader.

PS: Thanks for the kindness, I really appreciate it!

[FarazzShaikh]

Also, this usecase with MeshDepthMaterial seems to not require CSM. You can simply write a very small ShaderMaterial and assign that to this.frontDepthMaterial

[CosyStudios]

Ah, got it, thanks - so you basically check if csm_CustomDepth is set, and apply the patch if it is. I thought perhaps it worked a bit like #include statements do in three shaders..

I had originally just tried to create a new MeshDepthMaterial and use onBeforeCompile , but for some reason couldnt then access the uniforms later to update them, even after assigning them to userData in the onBeforeCompile method, but it was like onBeforeCompile never ran..

Agreed also that I could just use ShaderMaterial, I tried the following but also to no avail, in fact with errors in the glsl parts, none of these were reported nor setting gl_FragColor = vec4(0.0, 0.0, 0.0, 0.0); in the fragment shader for instance had no effect either- so for some reason or another my mesh.customDepthMaterial isnt being instantiated or honoured at all !

`

let depthShader = ShaderLib['depth']; let uniforms = UniformsUtils.clone(depthShader.uniforms);

  console.log('depth shader', depthShader, uniforms);

this.frontDepthMaterial = new ShaderMaterial({
    uniforms: {...UniformsUtils.clone(depthShader.uniforms), ...UniformsUtils.clone(Uniforms)},
    vertexShader: dkDepthVert,
    fragmentShader: depthShader.fragmentShader, // (or custom dkDepthFrag)
    transparent:true,
    depthPacking: RGBADepthPacking,
    side: FrontSide,
})

this.frontDepthMaterial.onBeforeCompile = (shader)=>{
    console.log("Depth: " +  shader);
};

this.mesh.customDepthMaterial = this.frontDepthMaterial;

`

[CosyStudios]

Sorry - cant for the love of me work out how to format code correctly in github !

[CosyStudios]

Ill report back if I get anything working at all !

I did build the mesh using this.frontDepthMaterial (& my custom frag shader) and it all compiled, throwing up a couple of glsl errors , which i then resolved but setting it as customDepthMaterial just doesnt seem to do anything , even if i intentially write some erroneous garbage in the shader so theres clearly something fundamentally wrong here regardless of the shader

[CosyStudios]

using three r.161 currently

[CosyStudios]

OK - so like an absolute tool, i had castShadow & receiveShadow set to false before so that my incomplete works didnt interfere with other shadows - seems these have to be true in order for custom depth material to be invoked at all.

Starting from scratch now ive confirmed i can actually make it work !! fool ..

[CosyStudios]

Hi Farazz - firstly apologies for stuffing this thread with my previous insanity.

Would you clarifying the usage of the patchmap keys?

My current working code looks like this:

In my vertex shader, im doing the calculations for the depth values - since this is where its done on the actual shader too. In it , I declare

varying float visibility; and set visibility appropiately based on sampling the depth video texture (where values less than 0.9 should be discarded in the frag shader)

`
this.frontDepthMaterial = new CustomShaderMaterial({
        uniforms: UniformsUtils.clone(Uniforms),
        baseMaterial: MeshDepthMaterial,
        vertexShader: dkDepthVert,
        //fragmentShader: (use the basematerial frag shader)
        patchMap: depthPatchMap,
        depthPacking: RGBADepthPacking,
        cacheKey: () => {
            return this.props.reference + '-depth-front'
        },
    })

`

Aim therefore is to set patch map to patch 2 entries in the original base depth shader...

const depthPatchMap =  {
  "*": {        // The keyword you will assign to in your custom shader
    "#if DEPTH_PACKING == 3200":
        ' // swapped out correct backtick as it mucks with the formatting here
        varying float visibility;
        #if DEPTH_PACKING == 3200
      '
  },
  visibility: {// The keyword you will assign to in your custom shader
    "float fragCoordZ = 0.5 * vHighPrecisionZW[0] / vHighPrecisionZW[1] + 0.5;":  
        '
        if ( visibility < 0.9) discard;
        float fragCoordZ = 0.5 * vHighPrecisionZW[0] / vHighPrecisionZW[1] + 0.5;
      '
  },
}

Ive also tried visibility in "" like the is but it doesnt seem to be patched in nor can i declare two lots of "" entries

Is the following valid?

const depthPatchMap =  {
  "*": {
    "#if DEPTH_PACKING == 3200":
        '
        varying float csm_CustomDepth;
        varying float visibility;
        #if DEPTH_PACKING == 3200
      ',
    "float fragCoordZ = 0.5 * vHighPrecisionZW[0] / vHighPrecisionZW[1] + 0.5;":
        '
        if ( visibility < 0.9) discard;
        float fragCoordZ = 0.5 * vHighPrecisionZW[0] / vHighPrecisionZW[1] + 0.5;
      '
  },
}

Seems to be , though im still not seeing any appropriate culling

[CosyStudios]

Just wanted to report back on progress..

I managed to get the depth material to work as a regular material for the model...

And the keyword replace works as advertised...

const depthPatchMap =  {
  "visibility": {
    "float fragCoordZ = 0.5 * vHighPrecisionZW[0] / vHighPrecisionZW[1] + 0.5;":
        `
        if ( visibility < 0.9) discard;
        float fragCoordZ = 0.5 * vHighPrecisionZW[0] / vHighPrecisionZW[1] + 0.5;
      `
  },
}

But, when I use it as a custom depth material, it still renders as the original plane geometry..

[CosyStudios]

.. In fact, ive done away with the patchmap all together. and just declared if ( visibility < 0.9) discard; in the frag shader...

removing that or altering the threshold to a lower value reveals the complete altered geometry

[FarazzShaikh]

Hello @CosyStudios, I’d love to assist you further, however GitHub issues is to track bugs and not support tickets. Please DM me on Discord @CantBeFaraz and we can pick up there

I’m not familiar with your use case so please try to create a working minimal reproduction of the issue on CodeSandbox or feel free to send me some code to work with

Lastly, I might not be available all the time so if you’d like to buy my time for more dedicated support feel free to also reach out through Discord or email

[CosyStudios]

Fair comment Farraz.. Github issues shouldnt be a sounding board for my internal processes I was hoping to solve this on my own steam with the intention of posting an actual solution in this thread for others as well as clean up the previous comments and hide the irrelevant findings - which i will still do.

I'll continue to crack at it but will follow on discord and attempt a sandbox should i need to reach out, happy to fund some time appropriately should I require a close eye.

Best..

[FarazzShaikh]

Hello,

I have decided that ShadowMaterial is advanced usage and supporting it does not provide enough utility to the average user to warrent complicating the codebase to support it OOTB.

ShadowMaterial will remain supported only though patchMaps using this patch map in perticular: https://github.com/FarazzShaikh/THREE-CustomShaderMaterial/issues/49#issuecomment-1997926041