haosulab / ManiSkill

SAPIEN Manipulation Skill Framework, a GPU parallelized robotics simulator and benchmark
https://maniskill.ai/
Apache License 2.0
827 stars 151 forks source link

Maniskill2 and ManiSkill2-learn on headless GPU nodes in HPC environment? #102

Open cousins opened 1 year ago

cousins commented 1 year ago

Hi,

I'm trying to help someone get ManiSkill2 and ManiSkill2-Learn running on our HPC system using GPU nodes. I assume that the general use of the GPUs is for processing but it also appears that there is some expectation of doing some rendering, using a X-Windows display. First, is there a way to decouple these? So that someone can do the GPU processing on a headless GPU node that has no rendering capabilities (we have A100 cards) and then use a different system to do the rendering?

I saw mention of the A100 card in your documentation so I was hopeful. The only JSON file in the /etc/vulkan directory is:

/etc/vulkan/implicit_layer.d/nvidia_layers.json

I saw some mention of running without a DISPLAY variable set so I tried the example:

Singularity> DISPLAY="" python -m mani_skill2.examples.demo_random_action -e PickCube-v0

opts: []
env_kwargs: {}
[2023-06-16 12:54:03.457] [svulkan2] [error] GLFW error: X11: Failed to open display 
[2023-06-16 12:54:03.457] [svulkan2] [warning] Continue without GLFW.
[2023-06-16 12:54:03.459] [svulkan2] [error] Some required Vulkan extension is not present. You may not use the renderer to render, however, CPU resources will be still available.
Segmentation fault (core dumped)

I have also looked through the Issues here on Github but it doesn't seem that the relevant issues have been resolved.

Can you please help me to figure out how to get this going?

Thanks,

Steve

xuanlinli17 commented 1 year ago

Hi Steve,

Could you ensure that the 3 jsons file exist according to https://haosulab.github.io/ManiSkill2/getting_started/installation.html#troubleshooting ?

cousins commented 1 year ago

Only the one JSON file that I mentioned was installed. I'm using the haosulab/mani-skill2 Docker container. Is there a container that is more up to date?

On Fri, Jun 16, 2023 at 4:50 PM Xuanlin (Simon) Li @.***> wrote:

Hi Steve,

Could you ensure that the 3 jsons file exist according to https://haosulab.github.io/ManiSkill2/getting_started/installation.html#troubleshooting ?

— Reply to this email directly, view it on GitHub https://github.com/haosulab/ManiSkill2/issues/102#issuecomment-1595303017, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDNXGKNVDELMTQYKT2EKWDXLTBJ7ANCNFSM6AAAAAAZJW2Y7M . You are receiving this because you authored the thread.Message ID: @.***>

--


Steve Cousins ​In​terim Director/Supercomputer Engineer Advanced Computing Group University of Maine System 244 Neville Hall (UMS Data Center) (207) 581-3574 Orono ME 04469 steve.cousins at maine.edu

xuanlinli17 commented 1 year ago

Probably you can build a docker here?

https://github.com/haosulab/ManiSkill2/blob/main/docker/Dockerfile

For the missing json files, they can be found here: https://github.com/haosulab/ManiSkill2/tree/main/docker, which will be added to the docker when you build it following the above dockerfile.

We'll also check the official docker for missing files.

xuanlinli17 commented 1 year ago

The official docker (docker pull haosulab/mani-skill2:latest) contains all 3 json files on my end.

/usr/share/vulkan/icd.d/nvidia_icd.json
/etc/vulkan/implicit_layer.d/nvidia_layers.json
/usr/share/glvnd/egl_vendor.d/10_nvidia.json
cousins commented 1 year ago

Hi Xuanlin,

I just tried and still only get the one nvidia_layers.json file:

singularity pull --disable-cache -F maniskill2-latest.simg docker:// ... sha256:d1f1ee3f64b3468bd53feaed80de74f13d91841cfbae380a0e4c15a506359032 2023/06/20 14:43:20 info unpack layer: sha256:92aaa5e43dd7c48efd447f3ff6d3352060c34c151a2c8da42ca72e90b4a81f1a INFO: Creating SIF file...

singularity shell maniskill2-latest.sing

Singularity> ls -al /etc/vulkan/* /etc/vulkan/explicit_layer.d: total 0 drwxr-xr-x 2 root root 3 Feb 10 2020 . drwxr-xr-x 5 root root 76 May 6 2022 ..

/etc/vulkan/icd.d: total 0 drwxr-xr-x 2 root root 3 Feb 10 2020 . drwxr-xr-x 5 root root 76 May 6 2022 ..

/etc/vulkan/implicit_layer.d: total 1 drwxr-xr-x 2 root root 41 Apr 2 20:38 . drwxr-xr-x 5 root root 76 May 6 2022 .. -rw-rw-r-- 1 root root 642 Mar 29 02:44 nvidia_layers.json

As you can see, I'm using Singularity to work with this container. We use this a lot and I have not had problems with it.

Any ideas?

Thanks,

Steve

On Sun, Jun 18, 2023 at 2:46 PM Xuanlin (Simon) Li @.***> wrote:

The official docker (docker pull haosulab/mani-skill2:latest) contains all 3 json files on my end.

— Reply to this email directly, view it on GitHub https://github.com/haosulab/ManiSkill2/issues/102#issuecomment-1596235295, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDNXGOVKHVNARIKQYXQ7Q3XL5EJ3ANCNFSM6AAAAAAZJW2Y7M . You are receiving this because you authored the thread.Message ID: @.***>

--


Steve Cousins ​In​terim Director/Supercomputer Engineer Advanced Computing Group University of Maine System 244 Neville Hall (UMS Data Center) (207) 581-3574 Orono ME 04469 steve.cousins at maine.edu

xuanlinli17 commented 1 year ago

Does

singularity pull mani-skill2-latest.sif docker://haosulab/mani-skill2:latest

work for you?

cousins commented 1 year ago

No. That's pretty much exactly what I did. I see now that the copy and paste didn't all get the whole command line. I just tried it again to be certain and I used the command:

singularity pull --disable-cache maniskill2-latest.sif docker://haosulab/mani-skill2:latest

And this is what I got with that image:

@.** ~]$ singularity shell maniskill2-latest.sif Singularity> ls -al /etc/vulkan/ /etc/vulkan/explicit_layer.d: total 0 drwxr-xr-x 2 root root 3 Feb 10 2020 . drwxr-xr-x 5 root root 76 May 6 2022 ..

/etc/vulkan/icd.d: total 0 drwxr-xr-x 2 root root 3 Feb 10 2020 . drwxr-xr-x 5 root root 76 May 6 2022 ..

/etc/vulkan/implicit_layer.d: total 1 drwxr-xr-x 2 root root 41 Apr 2 20:38 . drwxr-xr-x 5 root root 76 May 6 2022 .. -rw-rw-r-- 1 root root 642 Mar 29 02:44 nvidia_layers.json

I also tried creating the Docker image using the link you showed before but that Dockerfile expects the nvidia*.json files to be in the directory that is used to build the container. Since I don't have those files, it gave me an error and did not create the container.

Thanks very much for your help.

Best,

Steve

On Tue, Jun 20, 2023 at 4:13 PM Xuanlin (Simon) Li @.***> wrote:

Does

singularity pull maniskill2-latest.sif docker://haosulab/mani-skill2:latest

work for you?

— Reply to this email directly, view it on GitHub https://github.com/haosulab/ManiSkill2/issues/102#issuecomment-1599436631, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDNXGL43GVEAUULPMNW4ADXMH777ANCNFSM6AAAAAAZJW2Y7M . You are receiving this because you authored the thread.Message ID: @.***>

--


Steve Cousins ​In​terim Director/Supercomputer Engineer Advanced Computing Group University of Maine System 244 Neville Hall (UMS Data Center) (207) 581-3574 Orono ME 04469 steve.cousins at maine.edu

xuanlinli17 commented 1 year ago

The json files are here: https://github.com/haosulab/ManiSkill2/tree/main/docker

If you enter ManiSkill2/docker and then docker build . then it should add the json files.

JankowskiChristopher commented 6 months ago

@cousins were you able to solve your issue? I've just got the same error as you. I do not have sudo on the cluster and GPUs also are headless. I tried running the official Docker image through Singularity, but got the same errors as you. I asked our admin to install Vulkan, but he could not add the json config files as they would conflict with other cluster functionalities and without them and when not using the Docker image I get such error:

RuntimeError: vk::Instance::enumeratePhysicalDevices: ErrorInitializationFailed
StoneT2000 commented 6 months ago

Those json config files are absolutely required for rendering @JankowskiChristopher, there is currently no way around it. I am a little confused why those json config files can't be added however, what kind of conflicts are there?

JankowskiChristopher commented 6 months ago

Ok, I will try to ask our admin again, the previous answer I got from him was: "The additional NVIDIA drivers can be troublesome because they place files in directories managed by system packages. Each update of them will most likely result in their modification or removal. Therefore, for now, I haven't added them." (Sorry, maybe the word 'conflicts' was not correctly used here by me) But I guess that as updates of the cluster are not that frequent I will be able to finish the current project before any update. In the meantime I tried working with Singularity which is available on the cluster and correctly places the json config files, but ran into some issues #265 Do you think that a better option is to try with Singularity or ask our cluster admin for the json files? When I test vulkaninfo currently on the cluster I get such output (so I guess that Vulkan is correctly installed?)

WARNING: [Loader Message] Code 0 : terminator_CreateInstance: Failed to CreateInstance in ICD 4.  Skipping ICD.
'DISPLAY' environment variable not set... skipping surface info
==========
VULKANINFO
==========

Vulkan Instance Version: 1.3.204

Instance Extensions: count = 20
===============================
    VK_EXT_acquire_drm_display             : extension revision 1
    VK_EXT_acquire_xlib_display            : extension revision 1
    VK_EXT_debug_report                    : extension revision 10
    VK_EXT_debug_utils                     : extension revision 2
    VK_EXT_direct_mode_display             : extension revision 1
    VK_EXT_display_surface_counter         : extension revision 1
    VK_EXT_swapchain_colorspace            : extension revision 4
    VK_KHR_device_group_creation           : extension revision 1
    VK_KHR_display                         : extension revision 23
    VK_KHR_external_fence_capabilities     : extension revision 1
    VK_KHR_external_memory_capabilities    : extension revision 1
    VK_KHR_external_semaphore_capabilities : extension revision 1
    VK_KHR_get_display_properties2         : extension revision 1
    VK_KHR_get_physical_device_properties2 : extension revision 2
    VK_KHR_get_surface_capabilities2       : extension revision 1
    VK_KHR_surface                         : extension revision 25
    VK_KHR_surface_protected_capabilities  : extension revision 1
    VK_KHR_wayland_surface                 : extension revision 6
    VK_KHR_xcb_surface                     : extension revision 6
    VK_KHR_xlib_surface                    : extension revision 6

Layers: count = 3
=================
VK_LAYER_INTEL_nullhw (INTEL NULL HW) Vulkan version 1.1.73, layer version 1:
    Layer Extensions: count = 0
    Devices: count = 1
        GPU id = 0 (llvmpipe (LLVM 15.0.7, 256 bits))
        Layer-Device Extensions: count = 0

VK_LAYER_MESA_device_select (Linux device selection layer) Vulkan version 1.3.211, layer version 1:
    Layer Extensions: count = 0
    Devices: count = 1
        GPU id = 0 (llvmpipe (LLVM 15.0.7, 256 bits))
        Layer-Device Extensions: count = 0

VK_LAYER_MESA_overlay (Mesa Overlay layer) Vulkan version 1.3.211, layer version 1:
    Layer Extensions: count = 0
    Devices: count = 1
        GPU id = 0 (llvmpipe (LLVM 15.0.7, 256 bits))
        Layer-Device Extensions: count = 0

Device Groups:
==============
Group 0:
    Properties:
        physicalDevices: count = 1
            llvmpipe (LLVM 15.0.7, 256 bits) (ID: 0)
        subsetAllocation = 0

    Present Capabilities:
        llvmpipe (LLVM 15.0.7, 256 bits) (ID: 0):
            Can present images from the following devices: count = 1
                llvmpipe (LLVM 15.0.7, 256 bits) (ID: 0)
        Present modes: count = 1
            DEVICE_GROUP_PRESENT_MODE_LOCAL_BIT_KHR

Device Properties and Extensions:
=================================
GPU0:
VkPhysicalDeviceProperties:
---------------------------
    apiVersion        = 4206847 (1.3.255)
    driverVersion     = 1 (0x0001)
    vendorID          = 0x10005
    deviceID          = 0x0000
    deviceType        = PHYSICAL_DEVICE_TYPE_CPU
    deviceName        = llvmpipe (LLVM 15.0.7, 256 bits)
    pipelineCacheUUID = 32332e32-2e31-2d31-7562-756e7475332e

VkPhysicalDeviceLimits:
-----------------------
    maxImageDimension1D                             = 16384
    maxImageDimension2D                             = 16384
    maxImageDimension3D                             = 4096
    maxImageDimensionCube                           = 32768
    maxImageArrayLayers                             = 2048
    maxTexelBufferElements                          = 134217728
    maxUniformBufferRange                           = 65536
    maxStorageBufferRange                           = 134217728
    maxPushConstantsSize                            = 256
    maxMemoryAllocationCount                        = 4294967295
    maxSamplerAllocationCount                       = 32768
    bufferImageGranularity                          = 0x00000040
    sparseAddressSpaceSize                          = 0x00000000
    maxBoundDescriptorSets                          = 8
    maxPerStageDescriptorSamplers                   = 1000000
    maxPerStageDescriptorUniformBuffers             = 1000000
    maxPerStageDescriptorStorageBuffers             = 1000000
    maxPerStageDescriptorSampledImages              = 1000000
    maxPerStageDescriptorStorageImages              = 1000000
    maxPerStageDescriptorInputAttachments           = 1000000
    maxPerStageResources                            = 1000000
    maxDescriptorSetSamplers                        = 1000000
    maxDescriptorSetUniformBuffers                  = 1000000
    maxDescriptorSetUniformBuffersDynamic           = 1000000
    maxDescriptorSetStorageBuffers                  = 1000000
    maxDescriptorSetStorageBuffersDynamic           = 1000000
    maxDescriptorSetSampledImages                   = 1000000
    maxDescriptorSetStorageImages                   = 1000000
    maxDescriptorSetInputAttachments                = 1000000
    maxVertexInputAttributes                        = 32
    maxVertexInputBindings                          = 32
    maxVertexInputAttributeOffset                   = 2047
    maxVertexInputBindingStride                     = 2048
    maxVertexOutputComponents                       = 128
    maxTessellationGenerationLevel                  = 64
    maxTessellationPatchSize                        = 32
    maxTessellationControlPerVertexInputComponents  = 128
    maxTessellationControlPerVertexOutputComponents = 128
    maxTessellationControlPerPatchOutputComponents  = 128
    maxTessellationControlTotalOutputComponents     = 4096
    maxTessellationEvaluationInputComponents        = 128
    maxTessellationEvaluationOutputComponents       = 128
    maxGeometryShaderInvocations                    = 32
    maxGeometryInputComponents                      = 64
    maxGeometryOutputComponents                     = 128
    maxGeometryOutputVertices                       = 1024
    maxGeometryTotalOutputComponents                = 1024
    maxFragmentInputComponents                      = 128
    maxFragmentOutputAttachments                    = 8
    maxFragmentDualSrcAttachments                   = 2
    maxFragmentCombinedOutputResources              = 104
    maxComputeSharedMemorySize                      = 32768
    maxComputeWorkGroupCount: count = 3
        65535
        65535
        65535
    maxComputeWorkGroupInvocations                  = 1024
    maxComputeWorkGroupSize: count = 3
        1024
        1024
        1024
    subPixelPrecisionBits                           = 8
    subTexelPrecisionBits                           = 8
    mipmapPrecisionBits                             = 4
    maxDrawIndexedIndexValue                        = 4294967295
    maxDrawIndirectCount                            = 4294967295
    maxSamplerLodBias                               = 16
    maxSamplerAnisotropy                            = 16
    maxViewports                                    = 16
    maxViewportDimensions: count = 2
        16384
        16384
    viewportBoundsRange: count = 2
        -32768
        32768
    viewportSubPixelBits                            = 0
    minMemoryMapAlignment                           = 64
    minTexelBufferOffsetAlignment                   = 0x00000010
    minUniformBufferOffsetAlignment                 = 0x00000010
    minStorageBufferOffsetAlignment                 = 0x00000010
    minTexelOffset                                  = -32
    maxTexelOffset                                  = 31
    minTexelGatherOffset                            = -32
    maxTexelGatherOffset                            = 31
    minInterpolationOffset                          = -2
    maxInterpolationOffset                          = 2
    subPixelInterpolationOffsetBits                 = 8
    maxFramebufferWidth                             = 16384
    maxFramebufferHeight                            = 16384
    maxFramebufferLayers                            = 2048
    framebufferColorSampleCounts: count = 2
        SAMPLE_COUNT_1_BIT
        SAMPLE_COUNT_4_BIT
    framebufferDepthSampleCounts: count = 2
        SAMPLE_COUNT_1_BIT
        SAMPLE_COUNT_4_BIT
    framebufferStencilSampleCounts: count = 2
        SAMPLE_COUNT_1_BIT
        SAMPLE_COUNT_4_BIT
    framebufferNoAttachmentsSampleCounts: count = 2
        SAMPLE_COUNT_1_BIT
        SAMPLE_COUNT_4_BIT
    maxColorAttachments                             = 8
    sampledImageColorSampleCounts: count = 2
        SAMPLE_COUNT_1_BIT
        SAMPLE_COUNT_4_BIT
    sampledImageIntegerSampleCounts: count = 2
        SAMPLE_COUNT_1_BIT
        SAMPLE_COUNT_4_BIT
    sampledImageDepthSampleCounts: count = 2
        SAMPLE_COUNT_1_BIT
        SAMPLE_COUNT_4_BIT
    sampledImageStencilSampleCounts: count = 2
        SAMPLE_COUNT_1_BIT
        SAMPLE_COUNT_4_BIT
    storageImageSampleCounts: count = 2
        SAMPLE_COUNT_1_BIT
        SAMPLE_COUNT_4_BIT
    maxSampleMaskWords                              = 1
    timestampComputeAndGraphics                     = true
    timestampPeriod                                 = 1
    maxClipDistances                                = 8
    maxCullDistances                                = 8
    maxCombinedClipAndCullDistances                 = 8
    discreteQueuePriorities                         = 2
    pointSizeRange: count = 2
        0
        255
    lineWidthRange: count = 2
        1
        255
    pointSizeGranularity                            = 0.125
    lineWidthGranularity                            = 0.0078125
    strictLines                                     = true
    standardSampleLocations                         = true
    optimalBufferCopyOffsetAlignment                = 0x00000080
    optimalBufferCopyRowPitchAlignment              = 0x00000080
    nonCoherentAtomSize                             = 0x00000040

VkPhysicalDeviceSparseProperties:
---------------------------------
    residencyStandard2DBlockShape            = false
    residencyStandard2DMultisampleBlockShape = false
    residencyStandard3DBlockShape            = false
    residencyAlignedMipSize                  = false
    residencyNonResidentStrict               = false

VkPhysicalDeviceCustomBorderColorPropertiesEXT:
-----------------------------------------------
    maxCustomBorderColorSamplers = 32768

VkPhysicalDeviceDepthStencilResolveProperties:
----------------------------------------------
    supportedDepthResolveModes: count = 2
        RESOLVE_MODE_SAMPLE_ZERO_BIT
        RESOLVE_MODE_AVERAGE_BIT
    supportedStencilResolveModes: count = 1
        RESOLVE_MODE_SAMPLE_ZERO_BIT
    independentResolveNone = false
    independentResolve     = false

VkPhysicalDeviceDescriptorIndexingProperties:
---------------------------------------------
    maxUpdateAfterBindDescriptorsInAllPools              = 4294967295
    shaderUniformBufferArrayNonUniformIndexingNative     = true
    shaderSampledImageArrayNonUniformIndexingNative      = true
    shaderStorageBufferArrayNonUniformIndexingNative     = true
    shaderStorageImageArrayNonUniformIndexingNative      = true
    shaderInputAttachmentArrayNonUniformIndexingNative   = true
    robustBufferAccessUpdateAfterBind                    = true
    quadDivergentImplicitLod                             = true
    maxPerStageDescriptorUpdateAfterBindSamplers         = 1000000
    maxPerStageDescriptorUpdateAfterBindUniformBuffers   = 1000000
    maxPerStageDescriptorUpdateAfterBindStorageBuffers   = 1000000
    maxPerStageDescriptorUpdateAfterBindSampledImages    = 1000000
    maxPerStageDescriptorUpdateAfterBindStorageImages    = 1000000
    maxPerStageDescriptorUpdateAfterBindInputAttachments = 1000000
    maxPerStageUpdateAfterBindResources                  = 1000000
    maxDescriptorSetUpdateAfterBindSamplers              = 1000000
    maxDescriptorSetUpdateAfterBindUniformBuffers        = 1000000
    maxDescriptorSetUpdateAfterBindUniformBuffersDynamic = 1000000
    maxDescriptorSetUpdateAfterBindStorageBuffers        = 1000000
    maxDescriptorSetUpdateAfterBindStorageBuffersDynamic = 1000000
    maxDescriptorSetUpdateAfterBindSampledImages         = 1000000
    maxDescriptorSetUpdateAfterBindStorageImages         = 1000000
    maxDescriptorSetUpdateAfterBindInputAttachments      = 1000000

VkPhysicalDeviceDriverProperties:
---------------------------------
    driverID           = DRIVER_ID_MESA_LLVMPIPE
    driverName         = llvmpipe
    driverInfo         = Mesa 23.2.1-1ubuntu3.1~22.04.2 (LLVM 15.0.7)
    conformanceVersion = 1.3.1.1

VkPhysicalDeviceExternalMemoryHostPropertiesEXT:
------------------------------------------------
    minImportedHostPointerAlignment = 0x00001000

VkPhysicalDeviceFloatControlsProperties:
----------------------------------------
    denormBehaviorIndependence            = SHADER_FLOAT_CONTROLS_INDEPENDENCE_ALL
    roundingModeIndependence              = SHADER_FLOAT_CONTROLS_INDEPENDENCE_ALL
    shaderSignedZeroInfNanPreserveFloat16 = true
    shaderSignedZeroInfNanPreserveFloat32 = true
    shaderSignedZeroInfNanPreserveFloat64 = true
    shaderDenormPreserveFloat16           = false
    shaderDenormPreserveFloat32           = false
    shaderDenormPreserveFloat64           = false
    shaderDenormFlushToZeroFloat16        = false
    shaderDenormFlushToZeroFloat32        = false
    shaderDenormFlushToZeroFloat64        = false
    shaderRoundingModeRTEFloat16          = true
    shaderRoundingModeRTEFloat32          = true
    shaderRoundingModeRTEFloat64          = true
    shaderRoundingModeRTZFloat16          = false
    shaderRoundingModeRTZFloat32          = false
    shaderRoundingModeRTZFloat64          = false

VkPhysicalDeviceIDProperties:
-----------------------------
    deviceUUID      = 6d657361-3233-2e32-2e31-2d3175627500
    driverUUID      = 6c6c766d-7069-7065-5555-494400000000
    deviceNodeMask  = 0
    deviceLUIDValid = false

VkPhysicalDeviceInlineUniformBlockProperties:
---------------------------------------------
    maxInlineUniformBlockSize                               = 4096
    maxPerStageDescriptorInlineUniformBlocks                = 8
    maxPerStageDescriptorUpdateAfterBindInlineUniformBlocks = 8
    maxDescriptorSetInlineUniformBlocks                     = 8
    maxDescriptorSetUpdateAfterBindInlineUniformBlocks      = 8

VkPhysicalDeviceLineRasterizationPropertiesEXT:
-----------------------------------------------
    lineSubPixelPrecisionBits = 8

VkPhysicalDeviceMaintenance3Properties:
---------------------------------------
    maxPerSetDescriptors    = 1000000
    maxMemoryAllocationSize = 0x80000000

VkPhysicalDeviceMaintenance4Properties:
---------------------------------------
    maxBufferSize = 0xffffffff

VkPhysicalDeviceMultiDrawPropertiesEXT:
---------------------------------------
    maxMultiDrawCount = 2048

VkPhysicalDeviceMultiviewProperties:
------------------------------------
    maxMultiviewViewCount     = 6
    maxMultiviewInstanceIndex = 2147483647

VkPhysicalDevicePointClippingProperties:
----------------------------------------
    pointClippingBehavior = POINT_CLIPPING_BEHAVIOR_ALL_CLIP_PLANES

VkPhysicalDeviceProtectedMemoryProperties:
------------------------------------------
    protectedNoFault = false

VkPhysicalDeviceProvokingVertexPropertiesEXT:
---------------------------------------------
    provokingVertexModePerPipeline                       = true
    transformFeedbackPreservesTriangleFanProvokingVertex = true

VkPhysicalDevicePushDescriptorPropertiesKHR:
--------------------------------------------
    maxPushDescriptors = 32

VkPhysicalDeviceRobustness2PropertiesEXT:
-----------------------------------------
    robustStorageBufferAccessSizeAlignment = 0x00000001
    robustUniformBufferAccessSizeAlignment = 0x00000001

VkPhysicalDeviceSamplerFilterMinmaxProperties:
----------------------------------------------
    filterMinmaxSingleComponentFormats = true
    filterMinmaxImageComponentMapping  = true

VkPhysicalDeviceShaderIntegerDotProductProperties:
--------------------------------------------------
    integerDotProduct8BitUnsignedAccelerated                                      = false
    integerDotProduct8BitSignedAccelerated                                        = false
    integerDotProduct8BitMixedSignednessAccelerated                               = false
    integerDotProduct4x8BitPackedUnsignedAccelerated                              = false
    integerDotProduct4x8BitPackedSignedAccelerated                                = false
    integerDotProduct4x8BitPackedMixedSignednessAccelerated                       = false
    integerDotProduct16BitUnsignedAccelerated                                     = false
    integerDotProduct16BitSignedAccelerated                                       = false
    integerDotProduct16BitMixedSignednessAccelerated                              = false
    integerDotProduct32BitUnsignedAccelerated                                     = false
    integerDotProduct32BitSignedAccelerated                                       = false
    integerDotProduct32BitMixedSignednessAccelerated                              = false
    integerDotProduct64BitUnsignedAccelerated                                     = false
    integerDotProduct64BitSignedAccelerated                                       = false
    integerDotProduct64BitMixedSignednessAccelerated                              = false
    integerDotProductAccumulatingSaturating8BitUnsignedAccelerated                = false
    integerDotProductAccumulatingSaturating8BitSignedAccelerated                  = false
    integerDotProductAccumulatingSaturating8BitMixedSignednessAccelerated         = false
    integerDotProductAccumulatingSaturating4x8BitPackedUnsignedAccelerated        = false
    integerDotProductAccumulatingSaturating4x8BitPackedSignedAccelerated          = false
    integerDotProductAccumulatingSaturating4x8BitPackedMixedSignednessAccelerated = false
    integerDotProductAccumulatingSaturating16BitUnsignedAccelerated               = false
    integerDotProductAccumulatingSaturating16BitSignedAccelerated                 = false
    integerDotProductAccumulatingSaturating16BitMixedSignednessAccelerated        = false
    integerDotProductAccumulatingSaturating32BitUnsignedAccelerated               = false
    integerDotProductAccumulatingSaturating32BitSignedAccelerated                 = false
    integerDotProductAccumulatingSaturating32BitMixedSignednessAccelerated        = false
    integerDotProductAccumulatingSaturating64BitUnsignedAccelerated               = false
    integerDotProductAccumulatingSaturating64BitSignedAccelerated                 = false
    integerDotProductAccumulatingSaturating64BitMixedSignednessAccelerated        = false

VkPhysicalDeviceSubgroupProperties:
-----------------------------------
    subgroupSize              = 8
    supportedStages: count = 6
        SHADER_STAGE_FRAGMENT_BIT
        SHADER_STAGE_COMPUTE_BIT
        SHADER_STAGE_ALL_GRAPHICS
        SHADER_STAGE_ALL
        SHADER_STAGE_TASK_BIT_NV
        SHADER_STAGE_MESH_BIT_NV
    supportedOperations: count = 7
        SUBGROUP_FEATURE_BASIC_BIT
        SUBGROUP_FEATURE_VOTE_BIT
        SUBGROUP_FEATURE_ARITHMETIC_BIT
        SUBGROUP_FEATURE_BALLOT_BIT
        SUBGROUP_FEATURE_SHUFFLE_BIT
        SUBGROUP_FEATURE_SHUFFLE_RELATIVE_BIT
        SUBGROUP_FEATURE_QUAD_BIT
    quadOperationsInAllStages = false

VkPhysicalDeviceSubgroupSizeControlProperties:
----------------------------------------------
    minSubgroupSize              = 8
    maxSubgroupSize              = 8
    maxComputeWorkgroupSubgroups = 32
    requiredSubgroupSizeStages: count = 4
        SHADER_STAGE_FRAGMENT_BIT
        SHADER_STAGE_COMPUTE_BIT
        SHADER_STAGE_ALL_GRAPHICS
        SHADER_STAGE_ALL

VkPhysicalDeviceTexelBufferAlignmentProperties:
-----------------------------------------------
    storageTexelBufferOffsetAlignmentBytes       = 0x00000010
    storageTexelBufferOffsetSingleTexelAlignment = true
    uniformTexelBufferOffsetAlignmentBytes       = 0x00000010
    uniformTexelBufferOffsetSingleTexelAlignment = true

VkPhysicalDeviceTimelineSemaphoreProperties:
--------------------------------------------
    maxTimelineSemaphoreValueDifference = 18446744073709551615

VkPhysicalDeviceTransformFeedbackPropertiesEXT:
-----------------------------------------------
    maxTransformFeedbackStreams                = 4
    maxTransformFeedbackBuffers                = 4
    maxTransformFeedbackBufferSize             = 0xffffffff
    maxTransformFeedbackStreamDataSize         = 512
    maxTransformFeedbackBufferDataSize         = 512
    maxTransformFeedbackBufferDataStride       = 512
    transformFeedbackQueries                   = true
    transformFeedbackStreamsLinesTriangles     = false
    transformFeedbackRasterizationStreamSelect = false
    transformFeedbackDraw                      = true

VkPhysicalDeviceVertexAttributeDivisorPropertiesEXT:
----------------------------------------------------
    maxVertexAttribDivisor = 4294967295

VkPhysicalDeviceVulkan11Properties:
-----------------------------------
    deviceUUID                        = 6d657361-3233-2e32-2e31-2d3175627500
    driverUUID                        = 6c6c766d-7069-7065-5555-494400000000
    deviceNodeMask                    = 0
    deviceLUIDValid                   = false
    subgroupSize                      = 8
    subgroupSupportedStages: count = 6
        SHADER_STAGE_FRAGMENT_BIT
        SHADER_STAGE_COMPUTE_BIT
        SHADER_STAGE_ALL_GRAPHICS
        SHADER_STAGE_ALL
        SHADER_STAGE_TASK_BIT_NV
        SHADER_STAGE_MESH_BIT_NV
    subgroupSupportedOperations: count = 7
        SUBGROUP_FEATURE_BASIC_BIT
        SUBGROUP_FEATURE_VOTE_BIT
        SUBGROUP_FEATURE_ARITHMETIC_BIT
        SUBGROUP_FEATURE_BALLOT_BIT
        SUBGROUP_FEATURE_SHUFFLE_BIT
        SUBGROUP_FEATURE_SHUFFLE_RELATIVE_BIT
        SUBGROUP_FEATURE_QUAD_BIT
    subgroupQuadOperationsInAllStages = false
    pointClippingBehavior             = POINT_CLIPPING_BEHAVIOR_ALL_CLIP_PLANES
    maxMultiviewViewCount             = 6
    maxMultiviewInstanceIndex         = 2147483647
    protectedNoFault                  = false
    maxPerSetDescriptors              = 1000000
    maxMemoryAllocationSize           = 0x80000000

VkPhysicalDeviceVulkan12Properties:
-----------------------------------
    driverID                                             = DRIVER_ID_MESA_LLVMPIPE
    driverName                                           = llvmpipe
    driverInfo                                           = Mesa 23.2.1-1ubuntu3.1~22.04.2 (LLVM 15.0.7)
    conformanceVersion                                   = 1.3.1.1
    denormBehaviorIndependence                           = SHADER_FLOAT_CONTROLS_INDEPENDENCE_ALL
    roundingModeIndependence                             = SHADER_FLOAT_CONTROLS_INDEPENDENCE_ALL
    shaderSignedZeroInfNanPreserveFloat16                = true
    shaderSignedZeroInfNanPreserveFloat32                = true
    shaderSignedZeroInfNanPreserveFloat64                = true
    shaderDenormPreserveFloat16                          = false
    shaderDenormPreserveFloat32                          = false
    shaderDenormPreserveFloat64                          = false
    shaderDenormFlushToZeroFloat16                       = false
    shaderDenormFlushToZeroFloat32                       = false
    shaderDenormFlushToZeroFloat64                       = false
    shaderRoundingModeRTEFloat16                         = true
    shaderRoundingModeRTEFloat32                         = true
    shaderRoundingModeRTEFloat64                         = true
    shaderRoundingModeRTZFloat16                         = false
    shaderRoundingModeRTZFloat32                         = false
    shaderRoundingModeRTZFloat64                         = false
    maxUpdateAfterBindDescriptorsInAllPools              = 4294967295
    shaderUniformBufferArrayNonUniformIndexingNative     = true
    shaderSampledImageArrayNonUniformIndexingNative      = true
    shaderStorageBufferArrayNonUniformIndexingNative     = true
    shaderStorageImageArrayNonUniformIndexingNative      = true
    shaderInputAttachmentArrayNonUniformIndexingNative   = true
    robustBufferAccessUpdateAfterBind                    = true
    quadDivergentImplicitLod                             = true
    maxPerStageDescriptorUpdateAfterBindSamplers         = 1000000
    maxPerStageDescriptorUpdateAfterBindUniformBuffers   = 1000000
    maxPerStageDescriptorUpdateAfterBindStorageBuffers   = 1000000
    maxPerStageDescriptorUpdateAfterBindSampledImages    = 1000000
    maxPerStageDescriptorUpdateAfterBindStorageImages    = 1000000
    maxPerStageDescriptorUpdateAfterBindInputAttachments = 1000000
    maxPerStageUpdateAfterBindResources                  = 1000000
    maxDescriptorSetUpdateAfterBindSamplers              = 1000000
    maxDescriptorSetUpdateAfterBindUniformBuffers        = 1000000
    maxDescriptorSetUpdateAfterBindUniformBuffersDynamic = 1000000
    maxDescriptorSetUpdateAfterBindStorageBuffers        = 1000000
    maxDescriptorSetUpdateAfterBindStorageBuffersDynamic = 1000000
    maxDescriptorSetUpdateAfterBindSampledImages         = 1000000
    maxDescriptorSetUpdateAfterBindStorageImages         = 1000000
    maxDescriptorSetUpdateAfterBindInputAttachments      = 1000000
    supportedDepthResolveModes: count = 2
        RESOLVE_MODE_SAMPLE_ZERO_BIT
        RESOLVE_MODE_AVERAGE_BIT
    supportedStencilResolveModes: count = 1
        RESOLVE_MODE_SAMPLE_ZERO_BIT
    independentResolveNone                               = false
    independentResolve                                   = false
    filterMinmaxSingleComponentFormats                   = true
    filterMinmaxImageComponentMapping                    = true
    maxTimelineSemaphoreValueDifference                  = 18446744073709551615
    framebufferIntegerColorSampleCounts: count = 1
        SAMPLE_COUNT_1_BIT

VkPhysicalDeviceVulkan13Properties:
-----------------------------------
    minSubgroupSize                                                               = 8
    maxSubgroupSize                                                               = 8
    maxComputeWorkgroupSubgroups                                                  = 32
    requiredSubgroupSizeStages: count = 4
        SHADER_STAGE_FRAGMENT_BIT
        SHADER_STAGE_COMPUTE_BIT
        SHADER_STAGE_ALL_GRAPHICS
        SHADER_STAGE_ALL
    maxInlineUniformBlockSize                                                     = 4096
    maxPerStageDescriptorInlineUniformBlocks                                      = 8
    maxPerStageDescriptorUpdateAfterBindInlineUniformBlocks                       = 8
    maxDescriptorSetInlineUniformBlocks                                           = 8
    maxDescriptorSetUpdateAfterBindInlineUniformBlocks                            = 8
    maxInlineUniformTotalSize                                                     = 262144
    integerDotProduct8BitUnsignedAccelerated                                      = false
    integerDotProduct8BitSignedAccelerated                                        = false
    integerDotProduct8BitMixedSignednessAccelerated                               = false
    integerDotProduct4x8BitPackedUnsignedAccelerated                              = false
    integerDotProduct4x8BitPackedSignedAccelerated                                = false
    integerDotProduct4x8BitPackedMixedSignednessAccelerated                       = false
    integerDotProduct16BitUnsignedAccelerated                                     = false
    integerDotProduct16BitSignedAccelerated                                       = false
    integerDotProduct16BitMixedSignednessAccelerated                              = false
    integerDotProduct32BitUnsignedAccelerated                                     = false
    integerDotProduct32BitSignedAccelerated                                       = false
    integerDotProduct32BitMixedSignednessAccelerated                              = false
    integerDotProduct64BitUnsignedAccelerated                                     = false
    integerDotProduct64BitSignedAccelerated                                       = false
    integerDotProduct64BitMixedSignednessAccelerated                              = false
    integerDotProductAccumulatingSaturating8BitUnsignedAccelerated                = false
    integerDotProductAccumulatingSaturating8BitSignedAccelerated                  = false
    integerDotProductAccumulatingSaturating8BitMixedSignednessAccelerated         = false
    integerDotProductAccumulatingSaturating4x8BitPackedUnsignedAccelerated        = false
    integerDotProductAccumulatingSaturating4x8BitPackedSignedAccelerated          = false
    integerDotProductAccumulatingSaturating4x8BitPackedMixedSignednessAccelerated = false
    integerDotProductAccumulatingSaturating16BitUnsignedAccelerated               = false
    integerDotProductAccumulatingSaturating16BitSignedAccelerated                 = false
    integerDotProductAccumulatingSaturating16BitMixedSignednessAccelerated        = false
    integerDotProductAccumulatingSaturating32BitUnsignedAccelerated               = false
    integerDotProductAccumulatingSaturating32BitSignedAccelerated                 = false
    integerDotProductAccumulatingSaturating32BitMixedSignednessAccelerated        = false
    integerDotProductAccumulatingSaturating64BitUnsignedAccelerated               = false
    integerDotProductAccumulatingSaturating64BitSignedAccelerated                 = false
    integerDotProductAccumulatingSaturating64BitMixedSignednessAccelerated        = false
    storageTexelBufferOffsetAlignmentBytes                                        = 0x00000010
    storageTexelBufferOffsetSingleTexelAlignment                                  = true
    uniformTexelBufferOffsetAlignmentBytes                                        = 0x00000010
    uniformTexelBufferOffsetSingleTexelAlignment                                  = true
    maxBufferSize                                                                 = 0xffffffff

Device Extensions: count = 114
    VK_ARM_rasterization_order_attachment_access  : extension revision 1
    VK_EXT_4444_formats                           : extension revision 1
    VK_EXT_attachment_feedback_loop_dynamic_state : extension revision 1
    VK_EXT_attachment_feedback_loop_layout        : extension revision 2
    VK_EXT_border_color_swizzle                   : extension revision 1
    VK_EXT_calibrated_timestamps                  : extension revision 2
    VK_EXT_color_write_enable                     : extension revision 1
    VK_EXT_conditional_rendering                  : extension revision 2
    VK_EXT_custom_border_color                    : extension revision 12
    VK_EXT_depth_clip_control                     : extension revision 1
    VK_EXT_depth_clip_enable                      : extension revision 1
    VK_EXT_depth_range_unrestricted               : extension revision 1
    VK_EXT_descriptor_buffer                      : extension revision 1
    VK_EXT_descriptor_indexing                    : extension revision 2
    VK_EXT_dynamic_rendering_unused_attachments   : extension revision 1
    VK_EXT_extended_dynamic_state                 : extension revision 1
    VK_EXT_extended_dynamic_state2                : extension revision 1
    VK_EXT_extended_dynamic_state3                : extension revision 2
    VK_EXT_external_memory_host                   : extension revision 1
    VK_EXT_graphics_pipeline_library              : extension revision 1
    VK_EXT_host_query_reset                       : extension revision 1
    VK_EXT_image_2d_view_of_3d                    : extension revision 1
    VK_EXT_image_robustness                       : extension revision 1
    VK_EXT_image_sliced_view_of_3d                : extension revision 1
    VK_EXT_index_type_uint8                       : extension revision 1
    VK_EXT_inline_uniform_block                   : extension revision 1
    VK_EXT_line_rasterization                     : extension revision 1
    VK_EXT_memory_budget                          : extension revision 1
    VK_EXT_memory_priority                        : extension revision 1
    VK_EXT_mesh_shader                            : extension revision 1
    VK_EXT_multi_draw                             : extension revision 1
    VK_EXT_multisampled_render_to_single_sampled  : extension revision 1
    VK_EXT_mutable_descriptor_type                : extension revision 1
    VK_EXT_non_seamless_cube_map                  : extension revision 1
    VK_EXT_pageable_device_local_memory           : extension revision 1
    VK_EXT_pipeline_creation_cache_control        : extension revision 3
    VK_EXT_pipeline_creation_feedback             : extension revision 1
    VK_EXT_post_depth_coverage                    : extension revision 1
    VK_EXT_primitive_topology_list_restart        : extension revision 1
    VK_EXT_primitives_generated_query             : extension revision 1
    VK_EXT_private_data                           : extension revision 1
    VK_EXT_provoking_vertex                       : extension revision 1
    VK_EXT_rasterization_order_attachment_access  : extension revision 1
    VK_EXT_robustness2                            : extension revision 1
    VK_EXT_sampler_filter_minmax                  : extension revision 2
    VK_EXT_scalar_block_layout                    : extension revision 1
    VK_EXT_separate_stencil_usage                 : extension revision 1
    VK_EXT_shader_atomic_float                    : extension revision 1
    VK_EXT_shader_atomic_float2                   : extension revision 1
    VK_EXT_shader_demote_to_helper_invocation     : extension revision 1
    VK_EXT_shader_object                          : extension revision 1
    VK_EXT_shader_stencil_export                  : extension revision 1
    VK_EXT_shader_subgroup_ballot                 : extension revision 1
    VK_EXT_shader_subgroup_vote                   : extension revision 1
    VK_EXT_shader_viewport_index_layer            : extension revision 1
    VK_EXT_subgroup_size_control                  : extension revision 2
    VK_EXT_texel_buffer_alignment                 : extension revision 1
    VK_EXT_transform_feedback                     : extension revision 1
    VK_EXT_vertex_attribute_divisor               : extension revision 3
    VK_EXT_vertex_input_dynamic_state             : extension revision 2
    VK_GOOGLE_decorate_string                     : extension revision 1
    VK_GOOGLE_hlsl_functionality1                 : extension revision 1
    VK_KHR_16bit_storage                          : extension revision 1
    VK_KHR_8bit_storage                           : extension revision 1
    VK_KHR_bind_memory2                           : extension revision 1
    VK_KHR_buffer_device_address                  : extension revision 1
    VK_KHR_copy_commands2                         : extension revision 1
    VK_KHR_create_renderpass2                     : extension revision 1
    VK_KHR_dedicated_allocation                   : extension revision 3
    VK_KHR_depth_stencil_resolve                  : extension revision 1
    VK_KHR_descriptor_update_template             : extension revision 1
    VK_KHR_device_group                           : extension revision 4
    VK_KHR_draw_indirect_count                    : extension revision 1
    VK_KHR_driver_properties                      : extension revision 1
    VK_KHR_dynamic_rendering                      : extension revision 1
    VK_KHR_external_fence                         : extension revision 1
    VK_KHR_external_memory                        : extension revision 1
    VK_KHR_external_memory_fd                     : extension revision 1
    VK_KHR_external_semaphore                     : extension revision 1
    VK_KHR_format_feature_flags2                  : extension revision 2
    VK_KHR_get_memory_requirements2               : extension revision 1
    VK_KHR_image_format_list                      : extension revision 1
    VK_KHR_imageless_framebuffer                  : extension revision 1
    VK_KHR_incremental_present                    : extension revision 2
    VK_KHR_maintenance1                           : extension revision 2
    VK_KHR_maintenance2                           : extension revision 1
    VK_KHR_maintenance3                           : extension revision 1
    VK_KHR_maintenance4                           : extension revision 2
    VK_KHR_multiview                              : extension revision 1
    VK_KHR_pipeline_library                       : extension revision 1
    VK_KHR_push_descriptor                        : extension revision 2
    VK_KHR_relaxed_block_layout                   : extension revision 1
    VK_KHR_sampler_mirror_clamp_to_edge           : extension revision 3
    VK_KHR_separate_depth_stencil_layouts         : extension revision 1
    VK_KHR_shader_atomic_int64                    : extension revision 1
    VK_KHR_shader_clock                           : extension revision 1
    VK_KHR_shader_draw_parameters                 : extension revision 1
    VK_KHR_shader_float16_int8                    : extension revision 1
    VK_KHR_shader_float_controls                  : extension revision 4
    VK_KHR_shader_integer_dot_product             : extension revision 1
    VK_KHR_shader_non_semantic_info               : extension revision 1
    VK_KHR_shader_subgroup_extended_types         : extension revision 1
    VK_KHR_shader_terminate_invocation            : extension revision 1
    VK_KHR_spirv_1_4                              : extension revision 1
    VK_KHR_storage_buffer_storage_class           : extension revision 1
    VK_KHR_swapchain                              : extension revision 70
    VK_KHR_swapchain_mutable_format               : extension revision 1
    VK_KHR_synchronization2                       : extension revision 1
    VK_KHR_timeline_semaphore                     : extension revision 2
    VK_KHR_uniform_buffer_standard_layout         : extension revision 1
    VK_KHR_variable_pointers                      : extension revision 1
    VK_KHR_vulkan_memory_model                    : extension revision 3
    VK_KHR_zero_initialize_workgroup_memory       : extension revision 1
    VK_NV_device_generated_commands               : extension revision 3

VkQueueFamilyProperties:
========================
    queueProperties[0]:
    -------------------
        minImageTransferGranularity = (1,1,1)
        queueCount                  = 1
        queueFlags                  = QUEUE_GRAPHICS | QUEUE_COMPUTE | QUEUE_TRANSFER
        timestampValidBits          = 64
        present support             = false

VkPhysicalDeviceMemoryProperties:
=================================
memoryHeaps: count = 1
    memoryHeaps[0]:
        size   = 67378339840 (0xfb00fe000) (62.75 GiB)
        budget = 67378339840 (0xfb00fe000) (62.75 GiB)
        usage  = 9575428096 (0x23abd7000) (8.92 GiB)
        flags: count = 1
            MEMORY_HEAP_DEVICE_LOCAL_BIT
memoryTypes: count = 1
    memoryTypes[0]:
        heapIndex     = 0
        propertyFlags = 0x000f: count = 4
            MEMORY_PROPERTY_DEVICE_LOCAL_BIT
            MEMORY_PROPERTY_HOST_VISIBLE_BIT
            MEMORY_PROPERTY_HOST_COHERENT_BIT
            MEMORY_PROPERTY_HOST_CACHED_BIT
        usable for:
            IMAGE_TILING_OPTIMAL:
                color images
                FORMAT_D16_UNORM
                FORMAT_X8_D24_UNORM_PACK32
                FORMAT_D32_SFLOAT
                FORMAT_S8_UINT
                FORMAT_D24_UNORM_S8_UINT
                FORMAT_D32_SFLOAT_S8_UINT
                (non-sparse)
            IMAGE_TILING_LINEAR:
                color images
                (non-sparse)

VkPhysicalDeviceFeatures:
=========================
    robustBufferAccess                      = true
    fullDrawIndexUint32                     = true
    imageCubeArray                          = true
    independentBlend                        = true
    geometryShader                          = true
    tessellationShader                      = true
    sampleRateShading                       = true
    dualSrcBlend                            = true
    logicOp                                 = true
    multiDrawIndirect                       = true
    drawIndirectFirstInstance               = true
    depthClamp                              = true
    depthBiasClamp                          = true
    fillModeNonSolid                        = true
    depthBounds                             = false
    wideLines                               = true
    largePoints                             = true
    alphaToOne                              = true
    multiViewport                           = true
    samplerAnisotropy                       = true
    textureCompressionETC2                  = false
    textureCompressionASTC_LDR              = false
    textureCompressionBC                    = true
    occlusionQueryPrecise                   = true
    pipelineStatisticsQuery                 = true
    vertexPipelineStoresAndAtomics          = true
    fragmentStoresAndAtomics                = true
    shaderTessellationAndGeometryPointSize  = true
    shaderImageGatherExtended               = true
    shaderStorageImageExtendedFormats       = true
    shaderStorageImageMultisample           = true
    shaderStorageImageReadWithoutFormat     = true
    shaderStorageImageWriteWithoutFormat    = true
    shaderUniformBufferArrayDynamicIndexing = true
    shaderSampledImageArrayDynamicIndexing  = true
    shaderStorageBufferArrayDynamicIndexing = true
    shaderStorageImageArrayDynamicIndexing  = true
    shaderClipDistance                      = true
    shaderCullDistance                      = true
    shaderFloat64                           = true
    shaderInt64                             = true
    shaderInt16                             = true
    shaderResourceResidency                 = false
    shaderResourceMinLod                    = false
    sparseBinding                           = false
    sparseResidencyBuffer                   = false
    sparseResidencyImage2D                  = false
    sparseResidencyImage3D                  = false
    sparseResidency2Samples                 = false
    sparseResidency4Samples                 = false
    sparseResidency8Samples                 = false
    sparseResidency16Samples                = false
    sparseResidencyAliased                  = false
    variableMultisampleRate                 = false
    inheritedQueries                        = false

VkPhysicalDevice16BitStorageFeatures:
-------------------------------------
    storageBuffer16BitAccess           = true
    uniformAndStorageBuffer16BitAccess = true
    storagePushConstant16              = true
    storageInputOutput16               = false

VkPhysicalDevice4444FormatsFeaturesEXT:
---------------------------------------
    formatA4R4G4B4 = true
    formatA4B4G4R4 = true

VkPhysicalDevice8BitStorageFeatures:
------------------------------------
    storageBuffer8BitAccess           = true
    uniformAndStorageBuffer8BitAccess = true
    storagePushConstant8              = true

VkPhysicalDeviceBorderColorSwizzleFeaturesEXT:
----------------------------------------------
    borderColorSwizzle          = true
    borderColorSwizzleFromImage = true

VkPhysicalDeviceBufferDeviceAddressFeatures:
--------------------------------------------
    bufferDeviceAddress              = true
    bufferDeviceAddressCaptureReplay = false
    bufferDeviceAddressMultiDevice   = false

VkPhysicalDeviceColorWriteEnableFeaturesEXT:
--------------------------------------------
    colorWriteEnable = true

VkPhysicalDeviceConditionalRenderingFeaturesEXT:
------------------------------------------------
    conditionalRendering          = true
    inheritedConditionalRendering = false

VkPhysicalDeviceCustomBorderColorFeaturesEXT:
---------------------------------------------
    customBorderColors             = true
    customBorderColorWithoutFormat = true

VkPhysicalDeviceDepthClipControlFeaturesEXT:
--------------------------------------------
    depthClipControl = true

VkPhysicalDeviceDepthClipEnableFeaturesEXT:
-------------------------------------------
    depthClipEnable = true

VkPhysicalDeviceDescriptorIndexingFeatures:
-------------------------------------------
    shaderInputAttachmentArrayDynamicIndexing          = true
    shaderUniformTexelBufferArrayDynamicIndexing       = true
    shaderStorageTexelBufferArrayDynamicIndexing       = true
    shaderUniformBufferArrayNonUniformIndexing         = true
    shaderSampledImageArrayNonUniformIndexing          = true
    shaderStorageBufferArrayNonUniformIndexing         = true
    shaderStorageImageArrayNonUniformIndexing          = true
    shaderInputAttachmentArrayNonUniformIndexing       = true
    shaderUniformTexelBufferArrayNonUniformIndexing    = true
    shaderStorageTexelBufferArrayNonUniformIndexing    = true
    descriptorBindingUniformBufferUpdateAfterBind      = true
    descriptorBindingSampledImageUpdateAfterBind       = true
    descriptorBindingStorageImageUpdateAfterBind       = true
    descriptorBindingStorageBufferUpdateAfterBind      = true
    descriptorBindingUniformTexelBufferUpdateAfterBind = true
    descriptorBindingStorageTexelBufferUpdateAfterBind = true
    descriptorBindingUpdateUnusedWhilePending          = true
    descriptorBindingPartiallyBound                    = true
    descriptorBindingVariableDescriptorCount           = true
    runtimeDescriptorArray                             = true

VkPhysicalDeviceDynamicRenderingFeatures:
-----------------------------------------
    dynamicRendering = true

VkPhysicalDeviceExtendedDynamicState2FeaturesEXT:
-------------------------------------------------
    extendedDynamicState2                   = true
    extendedDynamicState2LogicOp            = true
    extendedDynamicState2PatchControlPoints = true

VkPhysicalDeviceExtendedDynamicStateFeaturesEXT:
------------------------------------------------
    extendedDynamicState = true

VkPhysicalDeviceHostQueryResetFeatures:
---------------------------------------
    hostQueryReset = true

VkPhysicalDeviceImageRobustnessFeatures:
----------------------------------------
    robustImageAccess = true

VkPhysicalDeviceImagelessFramebufferFeatures:
---------------------------------------------
    imagelessFramebuffer = true

VkPhysicalDeviceIndexTypeUint8FeaturesEXT:
------------------------------------------
    indexTypeUint8 = true

VkPhysicalDeviceInlineUniformBlockFeatures:
-------------------------------------------
    inlineUniformBlock                                 = true
    descriptorBindingInlineUniformBlockUpdateAfterBind = true

VkPhysicalDeviceLineRasterizationFeaturesEXT:
---------------------------------------------
    rectangularLines         = true
    bresenhamLines           = true
    smoothLines              = true
    stippledRectangularLines = true
    stippledBresenhamLines   = true
    stippledSmoothLines      = true

VkPhysicalDeviceMaintenance4Features:
-------------------------------------
    maintenance4 = true

VkPhysicalDeviceMemoryPriorityFeaturesEXT:
------------------------------------------
    memoryPriority = true

VkPhysicalDeviceMultiDrawFeaturesEXT:
-------------------------------------
    multiDraw = true

VkPhysicalDeviceMultiviewFeatures:
----------------------------------
    multiview                   = true
    multiviewGeometryShader     = true
    multiviewTessellationShader = true

VkPhysicalDevicePageableDeviceLocalMemoryFeaturesEXT:
-----------------------------------------------------
    pageableDeviceLocalMemory = true

VkPhysicalDevicePipelineCreationCacheControlFeatures:
-----------------------------------------------------
    pipelineCreationCacheControl = true

VkPhysicalDevicePrimitiveTopologyListRestartFeaturesEXT:
--------------------------------------------------------
    primitiveTopologyListRestart      = true
    primitiveTopologyPatchListRestart = true

VkPhysicalDevicePrivateDataFeatures:
------------------------------------
    privateData = true

VkPhysicalDeviceProtectedMemoryFeatures:
----------------------------------------
    protectedMemory = false

VkPhysicalDeviceProvokingVertexFeaturesEXT:
-------------------------------------------
    provokingVertexLast                       = true
    transformFeedbackPreservesProvokingVertex = true

VkPhysicalDeviceRobustness2FeaturesEXT:
---------------------------------------
    robustBufferAccess2 = true
    robustImageAccess2  = true
    nullDescriptor      = true

VkPhysicalDeviceSamplerYcbcrConversionFeatures:
-----------------------------------------------
    samplerYcbcrConversion = false

VkPhysicalDeviceScalarBlockLayoutFeatures:
------------------------------------------
    scalarBlockLayout = true

VkPhysicalDeviceSeparateDepthStencilLayoutsFeatures:
----------------------------------------------------
    separateDepthStencilLayouts = true

VkPhysicalDeviceShaderAtomicFloat2FeaturesEXT:
----------------------------------------------
    shaderBufferFloat16Atomics      = false
    shaderBufferFloat16AtomicAdd    = false
    shaderBufferFloat16AtomicMinMax = false
    shaderBufferFloat32AtomicMinMax = true
    shaderBufferFloat64AtomicMinMax = false
    shaderSharedFloat16Atomics      = false
    shaderSharedFloat16AtomicAdd    = false
    shaderSharedFloat16AtomicMinMax = false
    shaderSharedFloat32AtomicMinMax = true
    shaderSharedFloat64AtomicMinMax = false
    shaderImageFloat32AtomicMinMax  = true
    sparseImageFloat32AtomicMinMax  = false

VkPhysicalDeviceShaderAtomicFloatFeaturesEXT:
---------------------------------------------
    shaderBufferFloat32Atomics   = true
    shaderBufferFloat32AtomicAdd = true
    shaderBufferFloat64Atomics   = false
    shaderBufferFloat64AtomicAdd = false
    shaderSharedFloat32Atomics   = true
    shaderSharedFloat32AtomicAdd = true
    shaderSharedFloat64Atomics   = false
    shaderSharedFloat64AtomicAdd = false
    shaderImageFloat32Atomics    = true
    shaderImageFloat32AtomicAdd  = true
    sparseImageFloat32Atomics    = false
    sparseImageFloat32AtomicAdd  = false

VkPhysicalDeviceShaderAtomicInt64Features:
------------------------------------------
    shaderBufferInt64Atomics = true
    shaderSharedInt64Atomics = true

VkPhysicalDeviceShaderClockFeaturesKHR:
---------------------------------------
    shaderSubgroupClock = true
    shaderDeviceClock   = true

VkPhysicalDeviceShaderDemoteToHelperInvocationFeatures:
-------------------------------------------------------
    shaderDemoteToHelperInvocation = true

VkPhysicalDeviceShaderDrawParametersFeatures:
---------------------------------------------
    shaderDrawParameters = true

VkPhysicalDeviceShaderFloat16Int8Features:
------------------------------------------
    shaderFloat16 = true
    shaderInt8    = true

VkPhysicalDeviceShaderIntegerDotProductFeatures:
------------------------------------------------
    shaderIntegerDotProduct = true

VkPhysicalDeviceShaderSubgroupExtendedTypesFeatures:
----------------------------------------------------
    shaderSubgroupExtendedTypes = true

VkPhysicalDeviceShaderTerminateInvocationFeatures:
--------------------------------------------------
    shaderTerminateInvocation = true

VkPhysicalDeviceSubgroupSizeControlFeatures:
--------------------------------------------
    subgroupSizeControl  = true
    computeFullSubgroups = true

VkPhysicalDeviceSynchronization2Features:
-----------------------------------------
    synchronization2 = true

VkPhysicalDeviceTexelBufferAlignmentFeaturesEXT:
------------------------------------------------
    texelBufferAlignment = true

VkPhysicalDeviceTextureCompressionASTCHDRFeatures:
--------------------------------------------------
    textureCompressionASTC_HDR = false

VkPhysicalDeviceTimelineSemaphoreFeatures:
------------------------------------------
    timelineSemaphore = true

VkPhysicalDeviceTransformFeedbackFeaturesEXT:
---------------------------------------------
    transformFeedback = true
    geometryStreams   = true

VkPhysicalDeviceUniformBufferStandardLayoutFeatures:
----------------------------------------------------
    uniformBufferStandardLayout = true

VkPhysicalDeviceVariablePointersFeatures:
-----------------------------------------
    variablePointersStorageBuffer = true
    variablePointers              = true

VkPhysicalDeviceVertexAttributeDivisorFeaturesEXT:
--------------------------------------------------
    vertexAttributeInstanceRateDivisor     = true
    vertexAttributeInstanceRateZeroDivisor = true

VkPhysicalDeviceVertexInputDynamicStateFeaturesEXT:
---------------------------------------------------
    vertexInputDynamicState = true

VkPhysicalDeviceVulkan11Features:
---------------------------------
    storageBuffer16BitAccess           = true
    uniformAndStorageBuffer16BitAccess = true
    storagePushConstant16              = true
    storageInputOutput16               = false
    multiview                          = true
    multiviewGeometryShader            = true
    multiviewTessellationShader        = true
    variablePointersStorageBuffer      = true
    variablePointers                   = true
    protectedMemory                    = false
    samplerYcbcrConversion             = false
    shaderDrawParameters               = true

VkPhysicalDeviceVulkan12Features:
---------------------------------
    samplerMirrorClampToEdge                           = true
    drawIndirectCount                                  = true
    storageBuffer8BitAccess                            = true
    uniformAndStorageBuffer8BitAccess                  = true
    storagePushConstant8                               = true
    shaderBufferInt64Atomics                           = true
    shaderSharedInt64Atomics                           = true
    shaderFloat16                                      = true
    shaderInt8                                         = true
    descriptorIndexing                                 = true
    shaderInputAttachmentArrayDynamicIndexing          = true
    shaderUniformTexelBufferArrayDynamicIndexing       = true
    shaderStorageTexelBufferArrayDynamicIndexing       = true
    shaderUniformBufferArrayNonUniformIndexing         = true
    shaderSampledImageArrayNonUniformIndexing          = true
    shaderStorageBufferArrayNonUniformIndexing         = true
    shaderStorageImageArrayNonUniformIndexing          = true
    shaderInputAttachmentArrayNonUniformIndexing       = true
    shaderUniformTexelBufferArrayNonUniformIndexing    = true
    shaderStorageTexelBufferArrayNonUniformIndexing    = true
    descriptorBindingUniformBufferUpdateAfterBind      = true
    descriptorBindingSampledImageUpdateAfterBind       = true
    descriptorBindingStorageImageUpdateAfterBind       = true
    descriptorBindingStorageBufferUpdateAfterBind      = true
    descriptorBindingUniformTexelBufferUpdateAfterBind = true
    descriptorBindingStorageTexelBufferUpdateAfterBind = true
    descriptorBindingUpdateUnusedWhilePending          = true
    descriptorBindingPartiallyBound                    = true
    descriptorBindingVariableDescriptorCount           = true
    runtimeDescriptorArray                             = true
    samplerFilterMinmax                                = true
    scalarBlockLayout                                  = true
    imagelessFramebuffer                               = true
    uniformBufferStandardLayout                        = true
    shaderSubgroupExtendedTypes                        = true
    separateDepthStencilLayouts                        = true
    hostQueryReset                                     = true
    timelineSemaphore                                  = true
    bufferDeviceAddress                                = true
    bufferDeviceAddressCaptureReplay                   = false
    bufferDeviceAddressMultiDevice                     = false
    vulkanMemoryModel                                  = true
    vulkanMemoryModelDeviceScope                       = true
    vulkanMemoryModelAvailabilityVisibilityChains      = true
    shaderOutputViewportIndex                          = true
    shaderOutputLayer                                  = true
    subgroupBroadcastDynamicId                         = true

VkPhysicalDeviceVulkan13Features:
---------------------------------
    robustImageAccess                                  = true
    inlineUniformBlock                                 = true
    descriptorBindingInlineUniformBlockUpdateAfterBind = true
    pipelineCreationCacheControl                       = true
    privateData                                        = true
    shaderDemoteToHelperInvocation                     = true
    shaderTerminateInvocation                          = true
    subgroupSizeControl                                = true
    computeFullSubgroups                               = true
    synchronization2                                   = true
    textureCompressionASTC_HDR                         = false
    shaderZeroInitializeWorkgroupMemory                = true
    dynamicRendering                                   = true
    shaderIntegerDotProduct                            = true
    maintenance4                                       = true

VkPhysicalDeviceVulkanMemoryModelFeatures:
------------------------------------------
    vulkanMemoryModel                             = true
    vulkanMemoryModelDeviceScope                  = true
    vulkanMemoryModelAvailabilityVisibilityChains = true

VkPhysicalDeviceZeroInitializeWorkgroupMemoryFeatures:
------------------------------------------------------
    shaderZeroInitializeWorkgroupMemory = true