search

NVIDIA-AI-IOT / synthetic_data_generation_training_workflow

Workflow for generating synthetic data and training CV models.
Other
44 stars 8 forks source link

libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null) #1

Closed monajalal closed 9 months ago

monajalal commented 9 months ago

Hi Rishabh

I have Enterprise Omniverse and when I run this script an error happens with Nucleus. Could you please guide how to fix it?


(base) mona@ada:~/.local/share/ov/pkg/isaac_sim-2023.1.1$ ./python.sh standalone_palletjack_sdg.py --headless True --height 544 --width 960 --num_frames 2000 --distractors warehouse --data_dir /data/isaac-sim/

[18.921s] [ext: omni.isaac.sim.python-2023.1.1] startup
[18.922s] Simulation App Starting
2024-02-09 22:31:02 [18,920ms] [Warning] [omni.kit.widget.settings.settings_model] a value for setting /rtx/ecoMode/enabled has been requested but is not available
2024-02-09 22:31:02 [18,920ms] [Warning] [omni.kit.widget.settings.settings_model] a value for setting /rtx/directLighting/enabled has been requested but is not available
2024-02-09 22:31:02 [18,921ms] [Warning] [omni.kit.widget.settings.settings_model] a value for setting /rtx/reflections/enabled has been requested but is not available
2024-02-09 22:31:02 [18,921ms] [Warning] [omni.kit.widget.settings.settings_model] a value for setting /rtx/translucency/enabled has been requested but is not available
2024-02-09 22:31:02 [18,921ms] [Warning] [omni.kit.widget.settings.settings_model] a value for setting /rtx/raytracing/subsurface/enabled has been requested but is not available
2024-02-09 22:31:02 [18,921ms] [Warning] [omni.kit.widget.settings.settings_model] a value for setting /rtx/caustics/enabled has been requested but is not available
2024-02-09 22:31:06 [22,956ms] [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] duplicated action as SPACE (On Press) with omni.kit.widget.toolbar.omni.kit.widget.toolbar::toolbar::play!
2024-02-09 22:31:06 [22,956ms] [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] Cannot register Global [omni.kit.widget.toolbar] omni.kit.widget.toolbar::toolbar::play.SPACE, error code: Duplicated action definition
[24.457s] app ready
2024-02-09 22:31:08 [24,940ms] [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] duplicated action as SPACE (On Press) with omni.kit.widget.toolbar.omni.kit.widget.toolbar::toolbar::play!
2024-02-09 22:31:08 [24,940ms] [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] Cannot register Global [omni.kit.widget.toolbar] omni.kit.widget.toolbar::toolbar::play.SPACE, error code: Duplicated action definition
[25.201s] RTX ready
[25.543s] Simulation App Startup Complete
Loading Stage /Isaac/Environments/Simple_Warehouse/warehouse.usd
2024-02-09 22:31:28 [45,573ms] [Warning] [omni.client.python] Detected a blocking function. This will cause hitches or hangs in the UI. Please switch to the async version:
  File "/home/mona/.local/share/ov/pkg/isaac_sim-2023.1.1/standalone_palletjack_sdg.py", line 370, in <module>
  File "/home/mona/.local/share/ov/pkg/isaac_sim-2023.1.1/standalone_palletjack_sdg.py", line 272, in main
  File "/home/mona/.local/share/ov/pkg/isaac_sim-2023.1.1/standalone_palletjack_sdg.py", line 208, in prefix_with_isaac_asset_server
  File "/home/mona/.local/share/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/utils/nucleus.py", line 572, in get_assets_root_path
  File "/home/mona/.local/share/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/utils/nucleus.py", line 192, in check_server
  File "/home/mona/.local/share/ov/pkg/isaac_sim-2023.1.1/kit/extscore/omni.client/omni/client/__init__.py", line 158, in stat

Opening in existing browser session.
libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
monajalal commented 9 months ago

I also ran it in my local machine that has a non-enterprise omniverse and got this error:

(base) mona@DOS:~/.local/share/ov/pkg/isaac_sim-2023.1.1$ ./python.sh standalone_palletjack_sdg.py --headless True --height 544 --width 960 --num_frames 2000 --distractors warehouse --data_dir output_dir/
Warning: running in conda env, please deactivate before executing this script
If conda is desired please source setup_conda_env.sh in your python 3.10 conda env and run python normally
Starting kit application with the following args:  ['/home/mona/.local/share/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/home/mona/.local/share/ov/pkg/isaac_sim-2023.1.1/apps/omni.isaac.sim.python.kit', '--/app/tokens/exe-path=/home/mona/.local/share/ov/pkg/isaac_sim-2023.1.1/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True', '--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=960', '--/app/renderer/resolution/height=544', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/home/mona/.local/share/ov/pkg/isaac_sim-2023.1.1/exts', '--ext-folder', '/home/mona/.local/share/ov/pkg/isaac_sim-2023.1.1/apps', '--/physics/cudaDevice=0', '--portable', '--no-window']
Passing the following args to the base kit application:  ['--headless', 'True', '--height', '544', '--width', '960', '--num_frames', '2000', '--distractors', 'warehouse', '--data_dir', 'output_dir/']
[Info] [carb] Logging to file: /home/mona/.local/share/ov/pkg/isaac_sim-2023.1.1/kit/logs/Kit/Isaac-Sim/2023.1/kit_20240209_181835.log
2024-02-09 23:18:35 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[0.039s] [ext: omni.kit.async_engine-0.0.0] startup
[0.287s] [ext: omni.activity.core-1.0.1] startup
[0.289s] [ext: omni.assets.plugins-0.0.0] startup
[0.290s] [ext: omni.stats-0.0.0] startup
[0.290s] [ext: omni.client-1.0.1] startup
[0.295s] [ext: omni.activity.profiler-1.0.2] startup
[0.297s] [ext: omni.gpu_foundation-0.0.0] startup
[0.304s] [ext: omni.rtx.shadercache.vulkan-1.0.0] startup
[0.305s] [ext: carb.windowing.plugins-1.0.0] startup
[0.310s] [ext: omni.kit.renderer.init-0.0.0] startup
2024-02-09 23:20:00 [85,204ms] [Warning] [gpu.foundation.plugin] Skipping unsupported non-NVIDIA GPU: Intel(R) UHD Graphics (TGL GT1)
2024-02-09 23:20:00 [85,204ms] [Warning] [gpu.foundation.plugin] Skipping unsupported non-NVIDIA GPU: Intel(R) UHD Graphics (TGL GT1)

|---------------------------------------------------------------------------------------------|
| Driver Version: 5802005       | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | Intel(R) UHD Graphics (TGL GT1)  |        |     | 48010   MB | 8086      | 0          |
|     |                                  |        |     |            | 9a60      | 5518e060.. |
|     |                                  |        |     |            | 0         |            |
|=============================================================================================|
| OS: 22.04.2 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.2, Kernel: 6.5.0-17-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12101004 (1.21.1.4)
| Processor: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz | Cores: 8 | Logical: 16
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 64014 | Free Memory: 55931
| Total Page/Swap (MB): 975 | Free Page/Swap: 975
|---------------------------------------------------------------------------------------------|
2024-02-09 23:20:00 [85,205ms] [Error] [gpu.foundation.plugin] No device could be created. Some known system issues:
- The driver is not installed properly and requires a clean re-install.
- Your GPUs do not support RayTracing: DXR or Vulkan ray_tracing, or hardware is excluded due to performance.
- The driver cannot enumerate any GPU: driver, display, TCC mode or a docker issue. For Vulkan, test it with Vulkaninfo tool from Vulkan SDK, instead of nvidia-smi.
- For Ubuntu, it requires server-xorg-core 1.20.7+ and a display to work without --no-window.
- For Linux dockers, the setup is not complete. Install the latest driver, xServer and NVIDIA container runtime.

image

monajalal commented 9 months ago

update: https://forums.developer.nvidia.com/t/libva-error-vagetdrivernamebyindex-failed-with-unknown-libva-error-driver-name-null/282570

^ please check this if you get a chance and let me know why the GPU crashes?