search

isaac-sim / IsaacLab

Unified framework for robot learning built on NVIDIA Isaac Sim
https://isaac-sim.github.io/IsaacLab
Other
2.29k stars 937 forks source link

[Question] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0. #987

Closed LiangZhisama closed 1 month ago

LiangZhisama commented 2 months ago

Question

Hello everyone, I'm running Isaaclab on an ubuntu 20.04 remote server and installing via pip. Previously I was able to run and record videos and save them normally through command like ./isaaclab.sh -p source/standalone/workflows/rsl_rl/train.py --task Isaac-Reach-Franka-v0 --headless --video but after I removed and re-updated the Nvidia drivers, the following problem occurred when recording video:

[INFO] Using python from: /home/guardx/miniconda3/envs/isaaclab/bin/python                                                                                                        
[INFO][AppLauncher]: Loading experience file: /home/guardx/feihan_dev/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/app/../../../../../../apps/isaaclab.python.headless.rendering.kit
[Warning] [omni.isaac.kit.simulation_app] Modules: ['omni.kit_app'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
Starting kit application with the following args:  ['/home/guardx/miniconda3/envs/isaaclab/lib/python3.10/site-packages/isaacsim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/home/guardx/feihan_dev/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/app/../../../../../../apps/isaaclab.python.headless.rendering.kit', '--/app/tokens/exe-path=/home/guardx/miniconda3/envs/isaaclab/lib/python3.10/site-packages/omni', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True', '--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/home/guardx/miniconda3/envs/isaaclab/lib/python3.10/site-packages/isaacsim/exts', '--ext-folder', '/home/guardx/miniconda3/envs/isaaclab/lib/python3.10/site-packages/isaacsim/apps', '--/renderer/activeGpu=0', '--/physics/cudaDevice=0', '--portable', '--no-window', '--/app/window/hideUi=True']
Passing the following args to the base kit application:  ['--task', 'Isaac-Reach-Franka-v0', '--headless', '--video']
Loading user config located at: '/home/guardx/miniconda3/envs/isaaclab/lib/python3.10/site-packages/omni/data/Kit/Isaac-Sim/4.0/user.config.json'
[Info] [carb] Logging to file: /home/guardx/miniconda3/envs/isaaclab/lib/python3.10/site-packages/omni/logs/Kit/Isaac-Sim/4.0/kit_20240914_040852.log
2024-09-14 08:08:52 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[0.061s] [ext: omni.kit.async_engine-0.0.0] startup
[0.156s] [ext: omni.stats-1.0.1] startup
[0.157s] [ext: omni.client-1.1.0] startup
[0.189s] [ext: omni.datastore-0.0.0] startup
[0.190s] [ext: omni.blobkey-1.1.0] startup
[0.190s] [ext: omni.ujitso.default-1.0.0] startup
[0.191s] [ext: omni.hsscclient-0.0.0] startup
[0.191s] [ext: omni.rtx.shadercache.vulkan-1.0.0] startup
[0.192s] [ext: omni.assets.plugins-0.0.0] startup
[0.194s] [ext: omni.gpu_foundation-0.0.0] startup
[0.205s] [ext: carb.windowing.plugins-1.0.0] startup
2024-09-14 08:08:52 [178ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-14 08:08:52 [178ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[0.206s] [ext: omni.kit.renderer.init-0.0.0] startup
2024-09-14 08:08:52 [232ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Server version.
2024-09-14 08:08:52 [285ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-14 08:08:52 [285ms] [Error] [carb.graphics-vulkan.plugin] vkEnumeratePhysicalDevices failed. No physical device is found.
2024-09-14 08:08:52 [285ms] [Error] [carb.graphics-vulkan.plugin] No physical device is found.
2024-09-14 08:08:52 [286ms] [Error] [gpu.foundation.plugin] carb::graphics::createInstance failed.
2024-09-14 08:08:54 [2,070ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-14 08:08:54 [2,070ms] [Error] [carb.graphics-vulkan.plugin] vkEnumeratePhysicalDevices failed. No physical device is found.
2024-09-14 08:08:54 [2,070ms] [Error] [carb.graphics-vulkan.plugin] No physical device is found.
2024-09-14 08:08:54 [2,071ms] [Error] [gpu.foundation.plugin] carb::graphics::createInstance failed.
2024-09-14 08:08:55 [3,372ms] [Error] [omni.gpu_foundation_factory.plugin] Failed to create any GPU devices, including an attempt with compatibility mode.

(zenity:2450233): Gtk-WARNING **: 04:08:55.669: cannot open display: 
[3.447s] [ext: omni.kit.pipapi-0.0.0] startup
[3.449s] [ext: omni.kit.pip_archive-0.0.0] startup
[3.450s] [ext: omni.kit.telemetry-0.5.0] startup
[3.467s] [ext: omni.mtlx-0.1.0] startup
[3.468s] [ext: omni.usd.config-1.0.4] startup
[3.473s] [ext: omni.gpucompute.plugins-0.0.0] startup
[3.474s] [ext: omni.usd.libs-1.0.1] startup
[3.601s] [ext: omni.kit.loop-isaac-1.2.0] startup
[3.602s] [ext: omni.kit.test-0.0.0] startup
[3.644s] [ext: omni.appwindow-1.1.8] startup
2024-09-14 08:08:55 [3,617ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-14 08:08:55 [3,617ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[3.646s] [ext: omni.kit.renderer.core-1.0.1] startup
2024-09-14 08:08:55 [3,619ms] [Error] [omni.kit.renderer.plugin] GPU Foundation is not initialized!
2024-09-14 08:08:55 [3,622ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-14 08:08:55 [3,622ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[3.651s] [ext: omni.kit.renderer.capture-0.0.0] startup
[3.653s] [ext: omni.kit.renderer.imgui-1.0.1] startup
2024-09-14 08:08:55 [3,627ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-14 08:08:55 [3,627ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-09-14 08:08:55 [3,628ms] [Error] [omni.kit.imgui_renderer.plugin] _prepareDefaultWindow: invalid device attached to the default window!
[3.658s] [ext: omni.ui-2.23.11] startup
[3.668s] [ext: omni.kit.mainwindow-1.0.3] startup
[3.669s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[3.764s] [ext: omni.usd.schema.forcefield-106.0.20] startup
[3.798s] [ext: omni.usd.schema.anim-0.0.0] startup
[3.819s] [ext: omni.usd.schema.semantics-0.0.0] startup
[3.823s] [ext: omni.usd.schema.audio-0.0.0] startup
[3.826s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[3.833s] [ext: omni.usd.schema.physx-106.0.20] startup
[3.863s] [ext: omni.usd_resolver-1.0.0] startup
[3.868s] [ext: omni.kit.usd_undo-0.1.8] startup
[3.869s] [ext: omni.usd.core-1.2.11] startup
[3.873s] [ext: omni.timeline-1.0.10] startup
[3.874s] [ext: omni.kit.notification_manager-1.0.8] startup
[3.877s] [ext: omni.hydra.scene_delegate-0.3.3] startup
[3.884s] [ext: omni.kit.widget.context_menu-1.2.1] startup
[3.885s] [ext: omni.kit.widget.path_field-2.0.9] startup
[3.888s] [ext: carb.audio-0.1.0] startup
[3.892s] [ext: omni.kit.widget.browser_bar-2.0.10] startup
[3.893s] [ext: omni.kit.audiodeviceenum-1.0.1] startup
[3.894s] [ext: omni.activity.core-1.0.1] startup
[3.896s] [ext: omni.resourcemonitor-105.0.1] startup
[3.898s] [ext: omni.kit.window.popup_dialog-2.0.24] startup
[3.900s] [ext: omni.graph.exec-0.9.3] startup
[3.900s] [ext: omni.kit.actions.core-1.0.0] startup
[3.902s] [ext: usdrt.scenegraph-7.4.8] startup
[3.949s] [ext: omni.kit.widget.nucleus_connector-1.1.8] startup
[3.951s] [ext: omni.kit.exec.core-0.13.2] startup
[3.953s] [ext: omni.kit.commands-1.4.9] startup
[3.956s] [ext: omni.hydra.usdrt_delegate-7.4.7] startup
[3.977s] [ext: omni.kit.clipboard-1.0.3] startup
[3.978s] [ext: omni.usd-1.11.2] startup
[4.027s] [ext: omni.kit.widget.options_menu-1.1.4] startup
[4.030s] [ext: omni.kit.helper.file_utils-0.1.8] startup
[4.031s] [ext: omni.kit.widget.options_button-1.0.2] startup
[4.032s] [ext: omni.ui.scene-1.9.3] startup
[4.037s] [ext: omni.kit.widget.filebrowser-2.10.48] startup
[4.041s] [ext: omni.kit.usd.layers-2.1.31] startup
[4.049s] [ext: omni.kit.window.filepicker-2.10.34] startup
[4.060s] [ext: omni.kit.context_menu-1.8.0] startup
[4.067s] [ext: omni.kit.window.file_importer-1.1.11] startup
[4.070s] [ext: omni.kit.menu.utils-1.5.27] startup
[4.078s] [ext: omni.kit.hotkeys.core-1.3.3] startup
[4.079s] [ext: omni.ujitso.processor.texture-1.0.0] startup
[4.079s] [ext: omni.iray.libs-0.0.0] startup
[4.085s] [ext: omni.ujitso.client-0.0.0] startup
[4.086s] [ext: omni.volume-0.5.0] startup
[4.088s] [ext: omni.mdl.neuraylib-0.2.5] startup
[4.090s] [ext: omni.mdl-52.0.1] startup
[4.107s] [ext: omni.hydra.rtx-0.2.0] startup
[4.122s] [ext: omni.kit.material.library-1.4.4] startup
[4.126s] [ext: omni.hydra.engine.stats-1.0.2] startup
[4.130s] [ext: omni.kit.widget.settings-1.1.1] startup
[4.132s] [ext: omni.kit.hydra_texture-1.2.6] startup
[4.137s] [ext: omni.kit.viewport.registry-104.0.6] startup
[4.138s] [ext: omni.kit.widget.viewport-106.0.3] startup
[4.143s] [ext: omni.kit.raycast.query-1.0.5] startup
[4.152s] [ext: omni.kit.widget.filter-1.1.4] startup
[4.152s] [ext: omni.kit.viewport.window-106.0.8] startup
[4.169s] [ext: omni.kit.window.preferences-1.5.3] startup
[4.182s] [ext: omni.command.usd-1.0.3] startup
[4.183s] [ext: omni.kit.viewport.utility-1.0.17] startup
[4.183s] [ext: omni.kit.manipulator.transform-104.7.5] startup
[4.249s] [ext: omni.kit.window.file_exporter-1.0.29] startup
[4.252s] [ext: omni.inspect-1.0.1] startup
[4.256s] [ext: omni.graph.tools-1.78.0] startup
[4.284s] [ext: omni.graph.core-2.170.3] startup
[4.292s] [ext: omni.kit.widget.toolbar-1.6.2] startup
[4.303s] [ext: omni.graph-1.135.0] startup
[4.345s] [ext: omni.kit.manipulator.tool.snap-1.4.5] startup
[4.350s] [ext: omni.convexdecomposition-106.0.20] startup
[4.355s] [ext: omni.kit.widget.stage-2.10.26] startup
[4.365s] [ext: omni.kit.stage_template.core-1.1.21] startup
[4.367s] [ext: omni.kit.primitive.mesh-1.0.16] startup
[4.371s] [ext: omni.kvdb-106.0.20] startup
[4.373s] [ext: omni.physx.foundation-106.0.20] startup
[4.373s] [ext: omni.usdphysics-106.0.20] startup
[4.377s] [ext: omni.localcache-106.0.20] startup
[4.379s] [ext: omni.debugdraw-0.1.3] startup
[4.383s] [ext: omni.physx.cooking-106.0.20] startup
[4.396s] [ext: omni.kit.widget.searchfield-1.1.6] startup
[4.398s] [ext: omni.kit.widget.highlight_label-1.0.2] startup
[4.400s] [ext: omni.physx-106.0.20] startup
[4.415s] [ext: omni.kit.window.property-1.11.1] startup
[4.420s] [ext: omni.physx.commands-106.0.20] startup
[4.426s] [ext: omni.kit.property.usd-3.21.28] startup
[4.437s] [ext: omni.kit.stage_templates-1.2.3] startup
[4.442s] [ext: omni.kit.widget.text_editor-1.0.2] startup
[4.445s] [ext: omni.usdphysics.ui-106.0.20] startup
[4.474s] [ext: omni.physx.stageupdate-106.0.20] startup
[4.479s] [ext: omni.kit.manipulator.selector-1.1.1] startup
[4.484s] [ext: omni.physx.ui-106.0.20] startup
[4.515s] [ext: omni.kit.window.file-1.3.52] startup
[4.523s] [ext: omni.kit.widget.prompt-1.0.7] startup
[4.526s] [ext: omni.graph.action_core-1.1.4] startup
[4.536s] [ext: omni.kit.viewport.manipulator.transform-106.0.1] startup
[4.537s] [ext: omni.physx.demos-106.0.20] startup
[4.558s] [ext: omni.graph.ui-1.70.0] startup
[4.576s] [ext: omni.fabric.commands-1.1.4] startup
[4.584s] [ext: omni.kit.manipulator.prim.core-107.0.3] startup
[4.594s] [ext: omni.physx.graph-106.0.20] startup
[4.609s] [ext: omni.graph.action_nodes-1.23.0] startup
[4.620s] [ext: omni.graph.image.core-0.3.2] startup
[4.625s] [ext: omni.kit.manipulator.prim.fabric-106.0.1] startup
[4.629s] [ext: omni.kit.manipulator.prim.usd-106.0.1] startup
[4.633s] [ext: omni.graph.action-1.102.1] startup
[4.635s] [ext: omni.graph.image.nodes-1.0.2] startup
[4.638s] [ext: omni.kit.manipulator.selection-104.0.9] startup
[4.642s] [ext: omni.kit.manipulator.prim-106.0.0] startup
[4.644s] [ext: omni.graph.ui_nodes-1.25.1] startup
Warning: Possible version incompatibility. Attempting to load omni::fabric::IPath with version v0.2 against v0.1.
[4.653s] [ext: omni.graph.nodes-1.143.0] startup
Warning: Possible version incompatibility. Attempting to load omni::fabric::IPath with version v0.2 against v0.1.
[4.669s] [ext: omni.kit.window.toolbar-1.6.1] startup
[4.683s] [ext: omni.kit.widget.layers-1.7.9] startup
[4.732s] [ext: omni.physics.tensors-106.0.20] startup
[4.768s] [ext: omni.physx.supportui-106.0.20] startup
[4.843s] [ext: omni.kit.property.material-1.9.4] startup
[4.865s] [ext: omni.graph.bundle.action-2.0.4] startup
[4.868s] [ext: omni.physx.tensors-106.0.20] startup
[5.023s] [ext: omni.ui_query-1.1.2] startup
[5.061s] [ext: omni.kit.property.physx-106.0.20] startup
[5.146s] [ext: omni.physx.telemetry-106.0.20] startup
[5.154s] [ext: omni.kit.ui_test-1.2.18] startup
[5.159s] [ext: omni.physx.vehicle-106.0.20] startup
[5.178s] [ext: omni.kit.widget.graph-1.12.8] startup
[5.188s] [ext: omni.physx.cct-106.0.20] startup
[5.199s] [ext: omni.physx.camera-106.0.20] startup
[5.214s] [ext: omni.kit.graph.delegate.default-1.2.2] startup
[5.222s] [ext: omni.physx.bundle-106.0.20] startup
[5.223s] [ext: omni.sensors.tiled-0.0.4] startup
[5.228s] [ext: omni.kit.graph.editor.core-1.5.3] startup
[5.237s] [ext: omni.kit.graph.usd.commands-1.3.1] startup
[5.242s] [ext: omni.warp.core-1.2.1] startup
[5.332s] [ext: omni.kit.widget.material_preview-1.0.16] startup
[5.336s] [ext: omni.syntheticdata-0.6.7] startup
[5.361s] [ext: omni.isaac.version-1.1.0] startup
[5.366s] [ext: omni.warp-1.2.1] startup
[5.375s] [ext: omni.kit.window.material_graph-1.8.15] startup
[5.434s] [ext: omni.graph.scriptnode-1.19.1] startup
[5.443s] [ext: omni.isaac.nucleus-0.3.0] startup
[5.446s] [ext: omni.replicator.core-1.11.14] startup
2024-09-14 08:08:57 [5,653ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
Warp 1.2.1 initialized:
   CUDA Toolkit 11.8, Driver 12.4
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled)
     "cuda:1"   : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled)
     "cuda:2"   : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled)
     "cuda:3"   : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled)
   CUDA peer access:
     Supported fully (all-directional)
   Kernel cache:
     /home/guardx/.cache/warp/1.2.1
[6.186s] [ext: omni.isaac.core-3.18.1] startup
[7.731s] [ext: omni.isaac.kit-1.13.0] startup
[7.734s] [ext: omni.replicator.isaac-1.15.0] startup
[7.751s] [ext: omni.physx.fabric-106.0.20] startup
[7.762s] [ext: omni.kit.window.title-1.1.3] startup
2024-09-14 08:08:59 [7,737ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-14 08:08:59 [7,737ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-09-14 08:08:59 [7,739ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-14 08:08:59 [7,739ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[7.768s] [ext: omni.isaac.cloner-0.8.1] startup
[7.770s] [ext: omni.kit.viewport.rtx-104.0.1] startup
[7.770s] [ext: isaaclab.python.headless-1.0.0] startup
[7.773s] [ext: isaaclab.python.headless.rendering-1.0.0] startup
[7.781s] Simulation App Starting
2024-09-14 08:09:00 [8,082ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,088ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
[8.117s] app ready
2024-09-14 08:09:00 [8,247ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,262ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,270ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,279ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,287ms] [Warning] [rtx.neuraylib.plugin] [LINK:PLUGIN]   0.0   LINK   plug warn : Loading /home/guardx/miniconda3/envs/isaaclab/lib/python3.10/site-packages/isaacsim/extscache/omni.iray.libs/bin/iray/nvcuvid_video_decoder.so: libnvcuvid.so.1: cannot open shared object file: No such file or directory
2024-09-14 08:09:00 [8,287ms] [Warning] [rtx.neuraylib.plugin] [LINK:PLUGIN]   0.0   LINK   plug warn : Loading .\nt-x86-64\lib\nvcuvid_video_decoder.so: .\nt-x86-64\lib\nvcuvid_video_decoder.so: cannot open shared object file: No such file or directory
2024-09-14 08:09:00 [8,287ms] [Warning] [rtx.neuraylib.plugin] Failed to load nvcuvid h264 decoder plugin. Iray Bridge h264 support will not be available.
2024-09-14 08:09:00 [8,287ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,296ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,304ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,312ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,321ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,329ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,338ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,346ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.
2024-09-14 08:09:00 [8,354ms] [Warning] [omni.graph.core.plugin] OmniGraphSettings::getCudaDeviceOrdinal: unable to get a valid CUDA device id from the renderer. Defaulting to GPU0.

Then the program falls into an infinite loop showing this. Here is the nvidia drivers information for typing dpkg -l | grep nvidia

ii  libnvidia-cfg1-550-server:amd64            550.90.07-0lambda0.22.04.4                        amd64        NVIDIA binary OpenGL/GLX configuration library
rc  libnvidia-compute-550:i386                 550.107.02-0lambda0.22.04.2                       i386         NVIDIA libcompute package
ii  libnvidia-compute-550-server:amd64         550.90.07-0lambda0.22.04.4                        amd64        NVIDIA libcompute package
ii  libnvidia-fbc1-550:i386                    550.107.02-0lambda0.22.04.2                       i386         NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  nvidia-utils-510-server                    515.65.01-0lambda0~22.04.1                        amd64        Transitional package for nvidia-utils-515-server
ii  nvidia-utils-515-server                    520.61.05-0lambda0~22.04.1                        amd64        Transitional package for nvidia-utils-520-server
ii  nvidia-utils-520-server                    525.105.17-0lambda0.22.04.1                       amd64        Transitional package for nvidia-utils-525-server
ii  nvidia-utils-525-server                    535.183.01-0lambda0.22.04.1                       amd64        Transitional package for nvidia-utils-535-server
ii  nvidia-utils-535-server                    550.90.07-0lambda0.22.04.4                        amd64        Transitional package for nvidia-utils-550-server
ii  nvidia-utils-550-server                    550.90.07-0lambda0.22.04.4                        amd64        NVIDIA driver support binaries
ii  screen-resolution-extra                    0.18.2                                            all          Extension for the nvidia-settings control panel
ii  xserver-xorg-video-nvidia-550-server       550.90.07-0lambda0.22.04.4                        amd64        NVIDIA binary Xorg driver

I am so frustrated with this issue and would appreciate any help you can give me! Also let me know if you guys need any additional information.

Toni-SM commented 2 months ago

Hi @LiangZhisama

It seems to be GPU driver is not working. Use nvidia-smi to show GPU info

LiangZhisama commented 2 months ago

Hi @LiangZhisama

It seems to be GPU driver is not working. Use nvidia-smi to show GPU info

Thanks so much for your quick reply! Actually I can run the code and get normal output without turning on the video recording function(which may actually correspond to env.render()). And here is my GPU info: image And here is the info by running nvcc-V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0
LiangZhisama commented 2 months ago

Hi @LiangZhisama

It seems to be GPU driver is not working. Use nvidia-smi to show GPU info

Thanks for your answer! I notice that the updated NVIDIA driver version on remote server is incorrect, which is Xserver-xorg-video-nvidia-550-server. After I re-install the version nvidia-driver-550-server, problem solved.