vsgshadow without shadows on MacOS

[rainergericke]

I have tested the new shadowing functionality on Linux, Windows 11 and MacOS 14.0 (Sonoma). The Linux and Windows version of vsgshadow works well. On the Mac the shadows don't show up.

[robertosfield]

Could you run with Vulkan debug laeyer turned on to see that picks up on any issues?

vsgshadow -d

Does the vsgrendertotexture and vsgrendertotexturearray work fine?

[rainergericke]

vsghadow -d gives this info: createTestScene() extents = -0.5 -0.5 -1 4.99829 0.5 1

VUID-VkDescriptorImageInfo-mutableComparisonSamplers-04450(ERROR / SPEC): msgNum: 256275552 - Validation Error: [ VUID-VkDescriptorImageInfo-mutableComparisonSamplers-04450 ] | MessageID = 0xf467460 | vkUpdateDescriptorSets() (portability error): sampler comparison not available. The Vulkan spec states: If the VK_KHR_portability_subset extension is enabled, and VkPhysicalDevicePortabilitySubsetFeaturesKHR::mutableComparisonSamplers is VK_FALSE, then sampler must have been created with VkSamplerCreateInfo::compareEnable set to VK_FALSE (https://vulkan.lunarg.com/doc/view/1.3.261.1/mac/1.3-extensions/vkspec.html#VUID-VkDescriptorImageInfo-mutableComparisonSamplers-04450) Objects: 0 VUID-VkDescriptorImageInfo-mutableComparisonSamplers-04450(ERROR / SPEC): msgNum: 256275552 - Validation Error: [ VUID-VkDescriptorImageInfo-mutableComparisonSamplers-04450 ] | MessageID = 0xf467460 | vkUpdateDescriptorSets() (portability error): sampler comparison not available. The Vulkan spec states: If the VK_KHR_portability_subset extension is enabled, and VkPhysicalDevicePortabilitySubsetFeaturesKHR::mutableComparisonSamplers is VK_FALSE, then sampler must have been created with VkSamplerCreateInfo::compareEnable set to VK_FALSE (https://vulkan.lunarg.com/doc/view/1.3.261.1/mac/1.3-extensions/vkspec.html#VUID-VkDescriptorImageInfo-mutableComparisonSamplers-04450) Objects: 0 Average frame rate = 61.5228

vsgrendertotexture models/teapot.vsgt

and

vsgrendertotexturearray models/teapot.vsgt

work fine.

[robertosfield]

OK, this looks like a macOS specific issue. Could you run:

vsgshadow -a -d

And see what the calls are just prior to the error.

[rainergericke]

output of

vsgshadow -a -d

is in the attached file. vsgshadow_ad.txt

[robertosfield]

Thanks for the file. The ERROR message doesn't seem to match up with the setting used in the associated Sampler, so perhaps the ERROR message is a bit misleading and there is some feature that needs enabling or setting that isn't supported but at this point it seems like the ERROR message is complaining about something not being VK_TRUE while it is.

Have you used any Vulkan applications on macOS that use shadow maps that work?

[robertosfield]

Searching online, perhaps we need to set mutableComparisonSamplers to VK_TRUE in the PortabilitySubsetFeaturesKHR setup:

https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/VkPhysicalDevicePortabilitySubsetFeaturesKHR.html

[rainergericke]

Pure Vulkan examples can be found in Sasha Willem's repository https://github.com/SaschaWillems/Vulkan.git. Examples:

1. shadowmapping
2. shadowmappingomni
3. shadowmappingcascade

The examples can be built on the Mac and run well.

set mutableComparisonSamplers to VK_TRUE I could not find a reference to the rsp. structure in the VulkanSceneGraph sources. Can you give me a hint, where the setting shpuld be placed?

[robertosfield]

As Sasha's shadow mapping examples are working is encouraging as there will be a big overlap in what is being done.

Do these examples have any mutableComparisonSamplers setting? I will have to do some research about this portability extensions to figure out what might be needed.

[rainergericke]

I searched for mutableComparisonSamplers in the example Xcode project, it does not seem to be used anywhere.

[robertosfield]

On Sat, 7 Oct 2023 at 11:52, Rainer Gericke @.***> wrote:

I searched for mutableComparisonSamplers in the example Xcode project, it does not seem to be used anywhere.

Perhaps he sets up the image sampler differently. In Vulkan it's possible to declare an immutable sampler but I've never tried this.

Another possible differences is the settings I use for the sampler enable a PCF filter of shadow map, this helps a little with visual quality, perhaps Sasha's example does have this.

Message ID: @.***>

[rainergericke]

Here is an excerpt of Sascha's shadowmappingcascade.cpp (line 332 to 345) where the depth sampler is setup:

    // Shared sampler for cascade depth reads
    VkSamplerCreateInfo sampler = vks::initializers::samplerCreateInfo();
    sampler.magFilter = VK_FILTER_LINEAR;
    sampler.minFilter = VK_FILTER_LINEAR;
    sampler.mipmapMode = VK_SAMPLER_MIPMAP_MODE_LINEAR;
    sampler.addressModeU = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE;
    sampler.addressModeV = sampler.addressModeU;
    sampler.addressModeW = sampler.addressModeU;
    sampler.mipLodBias = 0.0f;
    sampler.maxAnisotropy = 1.0f;
    sampler.minLod = 0.0f;
    sampler.maxLod = 1.0f;
    sampler.borderColor = VK_BORDER_COLOR_FLOAT_OPAQUE_WHITE;
    VK_CHECK_RESULT(vkCreateSampler(device, &sampler, nullptr, &depth.sampler));

[robertosfield]

The code in ViewDependentState.cpp for setting you the shadow maps sampler is:

https://github.com/vsg-dev/VulkanSceneGraph/blob/master/src/vsg/state/ViewDependentState.cpp#L261

    // set up ShadowMaps
    auto shadowMapSampler = Sampler::create();
#define HARDWARE_PCF 1
#if HARDWARE_PCF == 1
    shadowMapSampler->minFilter = VK_FILTER_LINEAR;
    shadowMapSampler->magFilter = VK_FILTER_LINEAR;
    shadowMapSampler->mipmapMode = VK_SAMPLER_MIPMAP_MODE_LINEAR;
    shadowMapSampler->addressModeU = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE;
    shadowMapSampler->addressModeV = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE;
    shadowMapSampler->addressModeW = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE;
    shadowMapSampler->compareEnable = VK_TRUE;
    shadowMapSampler->compareOp = VK_COMPARE_OP_LESS;
#else
    shadowMapSampler->minFilter = VK_FILTER_NEAREST;
    shadowMapSampler->magFilter = VK_FILTER_NEAREST;
    shadowMapSampler->mipmapMode = VK_SAMPLER_MIPMAP_MODE_NEAREST;
    shadowMapSampler->addressModeU = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE;
    shadowMapSampler->addressModeV = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE;
    shadowMapSampler->addressModeW = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE;
#endif

You could try commenting out the lines:

shadowMapSampler->compareEnable = VK_TRUE;
shadowMapSampler->compareOp = VK_COMPARE_OP_LESS;

Or just setting HARDWARE_PCF to 0. The fragment shader also uses this as well so you'd need to edit the vsgExamples/data/shaders/standard_phong.frag, then use vsgshadow --custom to make sure it's use the custom shaders rather than the built-in ones.

If this works then at least we'd have an explanation as to what is failing and can start planning a solution,

[rainergericke]

I tried the #define HARDWARE_PCF 0 method as well as the comment out method. There was no difference. I left the vsgExamples/data/shaders/standard_phong.frag unchanged, since it has no preprocessor symbol HARDWARE_PCF defined. Did you mean something different?

[robertosfield]

On Sun, 8 Oct 2023 at 12:40, Rainer Gericke @.***> wrote:

I tried the #define HARDWARE_PCF 0 method as well as the comment out method. There was no difference.

Dang. Was pinning my hopes on that being a brakethrough.

I left the vsgExamples/data/shaders/standard_phong.frag unchanged, since it has no preprocessor symbol HARDWARE_PCF defined. Did you mean something different?

I must have removed the HARDWARE_PCF codepath from the shader after I finished debugging work.

Message ID: @.***>

[rainergericke]

Good news: the last commits for VSG let the shadows appear also on the Mac!

[robertosfield]

That's great news. Thanks to @timoore for the PR.