Amber return code when validation fails

[rayanht]

My use case for Amber is just to run SPIR-V shaders and dump the buffers to a file. In other words, I don't have any actual testing going on inside the Amber script.

In that scenario, when the Vulkan validation layer fails due to an improper SPIR-V assembly file, Amber exits with return code 0. I'm assuming this happens because there's technically not a hard failure as there are no tests to run. Is this expected behaviour?

Here's a sample Amber file to reproduce:

#!amber
SHADER compute shader SPIRV-ASM TARGET_ENV spv1.3
OpCapability Shader
OpMemoryModel Logical GLSL450
OpEntryPoint GLCompute %7 "main"
OpExecutionMode %7 LocalSize 1 1 1
OpDecorate %6 DescriptorSet 0
OpDecorate %6 Binding 0
%1 = OpTypeVoid
%2 = OpTypeFunction %1
%3 = OpTypeFloat 32
%4 = OpTypeInt 32 0
%5 = OpTypePointer StorageBuffer %4
%6 = OpVariable %5 StorageBuffer
%7 = OpFunction %1 DontInline %2
%8 = OpLabel
%9 = OpLoad %4 %6
%10 = OpIAdd %4 %9 %9
OpStore %6 %9
OpReturn
OpFunctionEnd
END
BUFFER buf0 DATA_TYPE uint32 STD430 DATA
76
END
PIPELINE compute pipeline
ATTACH shader
BIND BUFFER buf0 AS storage DESCRIPTOR_SET 0 BINDING 0
END
RUN pipeline 1 1 1

Running with bin/amber -t spv1.3 -v 1.2 tmp.amber, we get:

[ERROR] validation layer (Validation):
Validation Error: [ VUID-VkDeviceCreateInfo-pProperties-04451 ] Object 0: handle = 0x7fcdee5b1cd0, type = VK_OBJECT_TYPE_PHYSICAL_DEVICE; | MessageID = 0x3a3b6ca0 | vkCreateDevice: VK_KHR_portability_subset must be enabled because physical device VkPhysicalDevice 0x7fcdee5b1cd0[] supports it The Vulkan spec states: If the VK_KHR_portability_subset extension is included in pProperties of vkEnumerateDeviceExtensionProperties, ppEnabledExtensions must include "VK_KHR_portability_subset" (https://vulkan.lunarg.com/doc/view/1.2.198.1/mac/1.2-extensions/vkspec.html#VUID-VkDeviceCreateInfo-pProperties-04451)
[ERROR] validation layer (Validation):
Validation Error: [ UNASSIGNED-CoreValidation-Shader-InconsistentSpirv ] Object 0: handle = 0x7fcdef84ac18, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x6bbb14 | SPIR-V module not valid: StorageBuffer OpVariable <id> '2[%2]' has illegal type.
From Vulkan spec, section 14.5.2:
Variables identified with the StorageBuffer storage class are used to access transparent buffer backed resources. Such variables must be typed as OpTypeStruct, or an array of this type
  %2 = OpVariable %_ptr_StorageBuffer_uint StorageBuffer

Summary: 1 pass, 0 fail

And a return code of 0.

[paulthomson]

I am afraid this is working as intended. The Vulkan validation layers are not part of Amber, and they do not communicate their status to Amber. Their status is only communicated via logging to stderr (by default). As such, Amber cannot fail a test based on the validation layers.