Open HeegerGao opened 1 week ago
What driver version do you have (check nvidia-smi
)?
Thanks for your prompt reply. It's 535.104.05.
Tue Jul 2 02:31:35 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3090 On | 00000000:03:00.0 Off | N/A |
| 0% 41C P8 19W / 370W | 3MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 On | 00000000:04:00.0 On | N/A |
| 0% 35C P8 19W / 370W | 3MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
@fbxiang any idea?
If the machine has a Window display, maybe debugging with vkcube
could help (try and see if vkcube
can run under the NVIDIA GPU). If it is on a server. It would be a bit harder to debug. Maybe I can take a look at the output of sapien info --all
and vulkaninfo
.
Thank @fbxiang . Here are the outputs of sapien info --all
and vulkaninfo
. It seems like there are indeed some problems.
~$ sapien info --all
Traceback (most recent call last):
File "/home/gck/.conda/envs/contact/bin/sapien", line 58, in <module>
info += sapien.render.get_device_summary() + "\n"
RuntimeError: failed to find a rendering device
~$ vulkaninfo
ERROR: [Loader Message] Code 0 : libGLX_nvidia.so.0: cannot open shared object file: No such file or directory
'DISPLAY' environment variable not set... skipping surface info
error: XDG_RUNTIME_DIR not set in the environment.
error: XDG_RUNTIME_DIR not set in the environment.
error: XDG_RUNTIME_DIR not set in the environment.
error: XDG_RUNTIME_DIR not set in the environment.
error: XDG_RUNTIME_DIR not set in the environment.
WARNING: lavapipe is not a conformant vulkan implementation, testing use only.
==========
VULKANINFO
==========
Vulkan Instance Version: 1.2.131
Instance Extensions: count = 18
====================
VK_EXT_acquire_xlib_display : extension revision 1
VK_EXT_debug_report : extension revision 10
VK_EXT_debug_utils : extension revision 1
VK_EXT_direct_mode_display : extension revision 1
VK_EXT_display_surface_counter : extension revision 1
VK_KHR_device_group_creation : extension revision 1
VK_KHR_display : extension revision 23
VK_KHR_external_fence_capabilities : extension revision 1
VK_KHR_external_memory_capabilities : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2 : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2 : extension revision 1
VK_KHR_surface : extension revision 25
VK_KHR_surface_protected_capabilities : extension revision 1
VK_KHR_wayland_surface : extension revision 6
VK_KHR_xcb_surface : extension revision 6
VK_KHR_xlib_surface : extension revision 6
Layers: count = 4
=======
VK_LAYER_LUNARG_standard_validation (LunarG Standard Validation Layer) Vulkan version 1.0.131, layer version 1:
Layer Extensions: count = 0
Devices: count = 1
GPU id : 0 (llvmpipe (LLVM 12.0.0, 256 bits))
Layer-Device Extensions: count = 0
VK_LAYER_MESA_device_select (Linux device selection layer) Vulkan version 1.2.73, layer version 1:
Layer Extensions: count = 0
Devices: count = 1
GPU id : 0 (llvmpipe (LLVM 12.0.0, 256 bits))
Layer-Device Extensions: count = 0
VK_LAYER_MESA_overlay (Mesa Overlay layer) Vulkan version 1.1.73, layer version 1:
Layer Extensions: count = 0
Devices: count = 1
GPU id : 0 (llvmpipe (LLVM 12.0.0, 256 bits))
Layer-Device Extensions: count = 0
VK_LAYER_NV_optimus (NVIDIA Optimus layer) Vulkan version 1.2.155, layer version 1:
Layer Extensions: count = 0
Devices: count = 1
GPU id : 0 (llvmpipe (LLVM 12.0.0, 256 bits))
Layer-Device Extensions: count = 0
Presentable Surfaces:
=====================
error: XDG_RUNTIME_DIR not set in the environment.
error: XDG_RUNTIME_DIR not set in the environment.
error: XDG_RUNTIME_DIR not set in the environment.
error: XDG_RUNTIME_DIR not set in the environment.
Groups:
=======
Device Group Properties (Group 0):
physicalDeviceCount: count = 1
llvmpipe (LLVM 12.0.0, 256 bits) (ID: 0)
subsetAllocation = 0
Device Group Present Capabilities (Group 0):
WARNING: lavapipe is not a conformant vulkan implementation, testing use only.
llvmpipe (LLVM 12.0.0, 256 bits) (ID: 0)
Can present images from the following devices:
llvmpipe (LLVM 12.0.0, 256 bits) (ID: 0)
Present modes:
DEVICE_GROUP_PRESENT_MODE_LOCAL_BIT_KHR
Device Properties and Extensions:
=================================
GPU0:
VkPhysicalDeviceProperties:
---------------------------
apiVersion = 4198582 (1.1.182)
driverVersion = 1 (0x0001)
vendorID = 0x10005
deviceID = 0x0000
deviceType = PHYSICAL_DEVICE_TYPE_CPU
deviceName = llvmpipe (LLVM 12.0.0, 256 bits)
VkPhysicalDeviceLimits:
-----------------------
maxImageDimension1D = 16384
maxImageDimension2D = 16384
maxImageDimension3D = 4096
maxImageDimensionCube = 32768
maxImageArrayLayers = 2048
maxTexelBufferElements = 134217728
maxUniformBufferRange = 65536
maxStorageBufferRange = 134217728
maxPushConstantsSize = 128
maxMemoryAllocationCount = 4294967295
maxSamplerAllocationCount = 32768
bufferImageGranularity = 0x00000040
sparseAddressSpaceSize = 0x00000000
maxBoundDescriptorSets = 8
maxPerStageDescriptorSamplers = 32
maxPerStageDescriptorUniformBuffers = 15
maxPerStageDescriptorStorageBuffers = 16
maxPerStageDescriptorSampledImages = 128
maxPerStageDescriptorStorageImages = 16
maxPerStageDescriptorInputAttachments = 8
maxPerStageResources = 128
maxDescriptorSetSamplers = 32768
maxDescriptorSetUniformBuffers = 256
maxDescriptorSetUniformBuffersDynamic = 256
maxDescriptorSetStorageBuffers = 256
maxDescriptorSetStorageBuffersDynamic = 256
maxDescriptorSetSampledImages = 256
maxDescriptorSetStorageImages = 256
maxDescriptorSetInputAttachments = 256
maxVertexInputAttributes = 32
maxVertexInputBindings = 32
maxVertexInputAttributeOffset = 2047
maxVertexInputBindingStride = 2048
maxVertexOutputComponents = 128
maxTessellationGenerationLevel = 64
maxTessellationPatchSize = 32
maxTessellationControlPerVertexInputComponents = 128
maxTessellationControlPerVertexOutputComponents = 128
maxTessellationControlPerPatchOutputComponents = 128
maxTessellationControlTotalOutputComponents = 4096
maxTessellationEvaluationInputComponents = 128
maxTessellationEvaluationOutputComponents = 128
maxGeometryShaderInvocations = 32
maxGeometryInputComponents = 64
maxGeometryOutputComponents = 128
maxGeometryOutputVertices = 1024
maxGeometryTotalOutputComponents = 1024
maxFragmentInputComponents = 128
maxFragmentOutputAttachments = 8
maxFragmentDualSrcAttachments = 2
maxFragmentCombinedOutputResources = 8
maxComputeSharedMemorySize = 32768
maxComputeWorkGroupCount: count = 3
65535
65535
65535
maxComputeWorkGroupInvocations = 1024
maxComputeWorkGroupSize: count = 3
1024
1024
1024
subPixelPrecisionBits = 8
subTexelPrecisionBits = 8
mipmapPrecisionBits = 8
maxDrawIndexedIndexValue = 4294967295
maxDrawIndirectCount = 4294967295
maxSamplerLodBias = 16
maxSamplerAnisotropy = 16
maxViewports = 16
maxViewportDimensions: count = 2
16384
16384
viewportBoundsRange: count = 2
-32768
32768
viewportSubPixelBits = 0
minMemoryMapAlignment = 64
minTexelBufferOffsetAlignment = 0x00000010
minUniformBufferOffsetAlignment = 0x00000010
minStorageBufferOffsetAlignment = 0x00000010
minTexelOffset = -32
maxTexelOffset = 31
minTexelGatherOffset = -32
maxTexelGatherOffset = 31
minInterpolationOffset = -2
maxInterpolationOffset = 2
subPixelInterpolationOffsetBits = 8
maxFramebufferWidth = 16384
maxFramebufferHeight = 16384
maxFramebufferLayers = 2048
framebufferColorSampleCounts:
SAMPLE_COUNT_1_BIT
SAMPLE_COUNT_4_BIT
framebufferDepthSampleCounts:
SAMPLE_COUNT_1_BIT
SAMPLE_COUNT_4_BIT
framebufferStencilSampleCounts:
SAMPLE_COUNT_1_BIT
SAMPLE_COUNT_4_BIT
framebufferNoAttachmentsSampleCounts:
SAMPLE_COUNT_1_BIT
SAMPLE_COUNT_4_BIT
maxColorAttachments = 8
sampledImageColorSampleCounts:
SAMPLE_COUNT_1_BIT
SAMPLE_COUNT_4_BIT
sampledImageIntegerSampleCounts:
SAMPLE_COUNT_1_BIT
SAMPLE_COUNT_4_BIT
sampledImageDepthSampleCounts:
SAMPLE_COUNT_1_BIT
SAMPLE_COUNT_4_BIT
sampledImageStencilSampleCounts:
SAMPLE_COUNT_1_BIT
SAMPLE_COUNT_4_BIT
storageImageSampleCounts:
SAMPLE_COUNT_1_BIT
SAMPLE_COUNT_4_BIT
maxSampleMaskWords = 1
timestampComputeAndGraphics = true
timestampPeriod = 1
maxClipDistances = 8
maxCullDistances = 8
maxCombinedClipAndCullDistances = 8
discreteQueuePriorities = 2
pointSizeRange: count = 2
0
255
lineWidthRange: count = 2
1
255
pointSizeGranularity = 0.125
lineWidthGranularity = 0.0078125
strictLines = true
standardSampleLocations = true
optimalBufferCopyOffsetAlignment = 0x00000080
optimalBufferCopyRowPitchAlignment = 0x00000080
nonCoherentAtomSize = 0x00000040
VkPhysicalDeviceSparseProperties:
---------------------------------
residencyStandard2DBlockShape = false
residencyStandard2DMultisampleBlockShape = false
residencyStandard3DBlockShape = false
residencyAlignedMipSize = false
residencyNonResidentStrict = false
VkPhysicalDeviceDriverPropertiesKHR:
------------------------------------
driverID = UNKNOWN_VkDriverId
driverName = llvmpipe
driverInfo = Mesa 21.2.6 (LLVM 12.0.0)
conformanceVersion = 1.0.0.0
VkPhysicalDeviceIDProperties:
-----------------------------
deviceUUID = 00000000-0000-0000-0000-000000000000
driverUUID = 00000000-0000-0000-0000-000000000000
deviceNodeMask = 0
deviceLUIDValid = false
VkPhysicalDeviceLineRasterizationPropertiesEXT:
-----------------------------------------------
lineSubPixelPrecisionBits = 8
VkPhysicalDeviceMaintenance3Properties:
---------------------------------------
maxPerSetDescriptors = 1024
maxMemoryAllocationSize = 0x80000000
VkPhysicalDeviceMultiviewProperties:
------------------------------------
maxMultiviewViewCount = 6
maxMultiviewInstanceIndex = 2147483647
VkPhysicalDevicePointClippingProperties:
----------------------------------------
pointClippingBehavior = POINT_CLIPPING_BEHAVIOR_ALL_CLIP_PLANES
VkPhysicalDeviceProtectedMemoryProperties:
------------------------------------------
protectedNoFault = false
VkPhysicalDevicePushDescriptorPropertiesKHR:
--------------------------------------------
maxPushDescriptors = 32
VkPhysicalDeviceSamplerFilterMinmaxPropertiesEXT:
-------------------------------------------------
filterMinmaxSingleComponentFormats = true
filterMinmaxImageComponentMapping = true
VkPhysicalDeviceSubgroupProperties:
-----------------------------------
subgroupSize = 8
supportedStages:
SHADER_STAGE_FRAGMENT_BIT
SHADER_STAGE_COMPUTE_BIT
SHADER_STAGE_ALL_GRAPHICS
SHADER_STAGE_ALL
supportedOperations:
SUBGROUP_FEATURE_BASIC_BIT
SUBGROUP_FEATURE_VOTE_BIT
SUBGROUP_FEATURE_ARITHMETIC_BIT
SUBGROUP_FEATURE_BALLOT_BIT
quadOperationsInAllStages = false
VkPhysicalDeviceTransformFeedbackPropertiesEXT:
-----------------------------------------------
maxTransformFeedbackStreams = 4
maxTransformFeedbackBuffers = 4
maxTransformFeedbackBufferSize = 0xffffffff
maxTransformFeedbackStreamDataSize = 512
maxTransformFeedbackBufferDataSize = 512
maxTransformFeedbackBufferDataStride = 512
transformFeedbackQueries = true
transformFeedbackStreamsLinesTriangles = false
transformFeedbackRasterizationStreamSelect = false
transformFeedbackDraw = true
VkPhysicalDeviceVertexAttributeDivisorPropertiesEXT:
----------------------------------------------------
maxVertexAttribDivisor = 4294967295
Device Extensions: count = 54
------------------
VK_EXT_calibrated_timestamps : extension revision 2
VK_EXT_conditional_rendering : extension revision 2
VK_EXT_custom_border_color : extension revision 12
VK_EXT_extended_dynamic_state : extension revision 1
VK_EXT_extended_dynamic_state2 : extension revision 1
VK_EXT_host_query_reset : extension revision 1
VK_EXT_index_type_uint8 : extension revision 1
VK_EXT_line_rasterization : extension revision 1
VK_EXT_multi_draw : extension revision 1
VK_EXT_post_depth_coverage : extension revision 1
VK_EXT_private_data : extension revision 1
VK_EXT_provoking_vertex : extension revision 1
VK_EXT_sampler_filter_minmax : extension revision 2
VK_EXT_scalar_block_layout : extension revision 1
VK_EXT_separate_stencil_usage : extension revision 1
VK_EXT_shader_stencil_export : extension revision 1
VK_EXT_shader_viewport_index_layer : extension revision 1
VK_EXT_transform_feedback : extension revision 1
VK_EXT_vertex_attribute_divisor : extension revision 3
VK_EXT_vertex_input_dynamic_state : extension revision 2
VK_GOOGLE_decorate_string : extension revision 1
VK_GOOGLE_hlsl_functionality1 : extension revision 1
VK_KHR_16bit_storage : extension revision 1
VK_KHR_8bit_storage : extension revision 1
VK_KHR_bind_memory2 : extension revision 1
VK_KHR_buffer_device_address : extension revision 1
VK_KHR_copy_commands2 : extension revision 1
VK_KHR_create_renderpass2 : extension revision 1
VK_KHR_dedicated_allocation : extension revision 3
VK_KHR_descriptor_update_template : extension revision 1
VK_KHR_device_group : extension revision 4
VK_KHR_draw_indirect_count : extension revision 1
VK_KHR_driver_properties : extension revision 1
VK_KHR_external_fence : extension revision 1
VK_KHR_external_memory : extension revision 1
VK_KHR_external_semaphore : extension revision 1
VK_KHR_get_memory_requirements2 : extension revision 1
VK_KHR_image_format_list : extension revision 1
VK_KHR_imageless_framebuffer : extension revision 1
VK_KHR_incremental_present : extension revision 2
VK_KHR_maintenance1 : extension revision 2
VK_KHR_maintenance2 : extension revision 1
VK_KHR_maintenance3 : extension revision 1
VK_KHR_multiview : extension revision 1
VK_KHR_push_descriptor : extension revision 2
VK_KHR_relaxed_block_layout : extension revision 1
VK_KHR_sampler_mirror_clamp_to_edge : extension revision 3
VK_KHR_separate_depth_stencil_layouts : extension revision 1
VK_KHR_shader_atomic_int64 : extension revision 1
VK_KHR_shader_draw_parameters : extension revision 1
VK_KHR_storage_buffer_storage_class : extension revision 1
VK_KHR_swapchain : extension revision 70
VK_KHR_uniform_buffer_standard_layout : extension revision 1
VK_KHR_variable_pointers : extension revision 1
VkQueueFamilyProperties:
========================
queueProperties[0]:
------------------
minImageTransferGranularity = (1,1,1)
queueCount = 1
queueFlags = QUEUE_GRAPHICS | QUEUE_COMPUTE | QUEUE_TRANSFER
timestampValidBits = 64
present support = false
VkPhysicalDeviceMemoryProperties:
=================================
memoryHeaps: count = 1
memoryHeaps[0]:
size = 2147483648 (0x80000000) (2.00 GiB)
budget = 0
usage = 0
flags:
MEMORY_HEAP_DEVICE_LOCAL_BIT
memoryTypes: count = 1
memoryTypes[0]:
heapIndex = 0
propertyFlags = 0x000f:
MEMORY_PROPERTY_DEVICE_LOCAL_BIT
MEMORY_PROPERTY_HOST_VISIBLE_BIT
MEMORY_PROPERTY_HOST_COHERENT_BIT
MEMORY_PROPERTY_HOST_CACHED_BIT
usable for:
IMAGE_TILING_OPTIMAL: color images, FORMAT_D16_UNORM, FORMAT_X8_D24_UNORM_PACK32, FORMAT_D32_SFLOAT, FORMAT_S8_UINT, FORMAT_D24_UNORM_S8_UINT, FORMAT_D32_SFLOAT_S8_UINT
IMAGE_TILING_LINEAR: color images
VkPhysicalDeviceFeatures:
=========================
robustBufferAccess = true
fullDrawIndexUint32 = true
imageCubeArray = true
independentBlend = true
geometryShader = true
tessellationShader = true
sampleRateShading = true
dualSrcBlend = true
logicOp = true
multiDrawIndirect = true
drawIndirectFirstInstance = true
depthClamp = true
depthBiasClamp = true
fillModeNonSolid = true
depthBounds = false
wideLines = true
largePoints = true
alphaToOne = true
multiViewport = true
samplerAnisotropy = false
textureCompressionETC2 = false
textureCompressionASTC_LDR = false
textureCompressionBC = true
occlusionQueryPrecise = true
pipelineStatisticsQuery = true
vertexPipelineStoresAndAtomics = true
fragmentStoresAndAtomics = true
shaderTessellationAndGeometryPointSize = true
shaderImageGatherExtended = true
shaderStorageImageExtendedFormats = true
shaderStorageImageMultisample = true
shaderStorageImageReadWithoutFormat = false
shaderStorageImageWriteWithoutFormat = true
shaderUniformBufferArrayDynamicIndexing = false
shaderSampledImageArrayDynamicIndexing = false
shaderStorageBufferArrayDynamicIndexing = false
shaderStorageImageArrayDynamicIndexing = false
shaderClipDistance = true
shaderCullDistance = true
shaderFloat64 = true
shaderInt64 = true
shaderInt16 = true
shaderResourceResidency = false
shaderResourceMinLod = false
sparseBinding = false
sparseResidencyBuffer = false
sparseResidencyImage2D = false
sparseResidencyImage3D = false
sparseResidency2Samples = false
sparseResidency4Samples = false
sparseResidency8Samples = false
sparseResidency16Samples = false
sparseResidencyAliased = false
variableMultisampleRate = false
inheritedQueries = false
VkPhysicalDevice16BitStorageFeatures:
-------------------------------------
storageBuffer16BitAccess = true
uniformAndStorageBuffer16BitAccess = true
storagePushConstant16 = true
storageInputOutput16 = false
VkPhysicalDevice8BitStorageFeaturesKHR:
---------------------------------------
storageBuffer8BitAccess = true
uniformAndStorageBuffer8BitAccess = true
storagePushConstant8 = true
VkPhysicalDeviceBufferDeviceAddressFeaturesKHR:
-----------------------------------------------
bufferDeviceAddress = true
bufferDeviceAddressCaptureReplay = false
bufferDeviceAddressMultiDevice = false
VkPhysicalDeviceConditionalRenderingFeaturesEXT:
------------------------------------------------
conditionalRendering = true
inheritedConditionalRendering = false
VkPhysicalDeviceHostQueryResetFeaturesEXT:
------------------------------------------
hostQueryReset = true
VkPhysicalDeviceImagelessFramebufferFeaturesKHR:
------------------------------------------------
imagelessFramebuffer = true
VkPhysicalDeviceIndexTypeUint8FeaturesEXT:
------------------------------------------
indexTypeUint8 = true
VkPhysicalDeviceLineRasterizationFeaturesEXT:
---------------------------------------------
rectangularLines = true
bresenhamLines = true
smoothLines = true
stippledRectangularLines = true
stippledBresenhamLines = true
stippledSmoothLines = true
VkPhysicalDeviceMultiviewFeatures:
----------------------------------
multiview = true
multiviewGeometryShader = true
multiviewTessellationShader = true
VkPhysicalDeviceProtectedMemoryFeatures:
----------------------------------------
protectedMemory = false
VkPhysicalDeviceSamplerYcbcrConversionFeatures:
-----------------------------------------------
samplerYcbcrConversion = false
VkPhysicalDeviceScalarBlockLayoutFeaturesEXT:
---------------------------------------------
scalarBlockLayout = true
VkPhysicalDeviceSeparateDepthStencilLayoutsFeaturesKHR:
-------------------------------------------------------
separateDepthStencilLayouts = true
VkPhysicalDeviceShaderAtomicInt64FeaturesKHR:
---------------------------------------------
shaderBufferInt64Atomics = true
shaderSharedInt64Atomics = true
VkPhysicalDeviceShaderDrawParametersFeatures:
---------------------------------------------
shaderDrawParameters = true
VkPhysicalDeviceTransformFeedbackFeaturesEXT:
---------------------------------------------
transformFeedback = true
geometryStreams = true
VkPhysicalDeviceUniformBufferStandardLayoutFeaturesKHR:
-------------------------------------------------------
uniformBufferStandardLayout = true
VkPhysicalDeviceVariablePointersFeatures:
-----------------------------------------
variablePointersStorageBuffer = true
variablePointers = false
VkPhysicalDeviceVertexAttributeDivisorFeaturesEXT:
--------------------------------------------------
vertexAttributeInstanceRateDivisor = true
vertexAttributeInstanceRateZeroDivisor = false
libGLX_nvidia.so.0 is NVIDIA's graphics driver and it is required for any kind of rendering. Vulkan info shows that your NVIDIA GPU is not capable of rendering because you are missing this driver. It can probably be fixed by re-installing NVIDIA driver (if you use apt, make sure you install nvidia-driver-xxx without the word server or headless in its name).
Hi, I am installing ManiSkill on my headless server. After installing mani_skill and torch with pip, I run
python -m mani_skill.examples.demo_random_action
and then I get the following error:I am using Ubuntu 20.04, with 2 3090 GPUs. CUDA version is 11.7. I am using python 3.9. I am sure the three vulkan json files exist.