[bug] Isaac lab streaming not working with client 103.1.1

ben-kaye commented 2 weeks ago

Describe the bug

omni.isaac.lab.app.AppLauncher does not work with livestream=1 using container

Steps to reproduce

run any Isaac lab script eg. ./isaaclab.sh -p source/standalone/tutorials/00_sim/create_empty.py --livestream 1 attempt to connect via streaming client fails unknown error 0x100

<ON HOST>
/workspace/isaaclab# ./isaaclab.sh -p source/standalone/tutorials/00_sim/create_empty.py --livestream 1
...
[5.919s] Simulation App Starting
[9.191s] app ready
2024-09-06 13:40:35 [9,560ms] [Warning] [omni.kit.imgui_renderer.plugin] _createExtendCursor: No windowing.
2024-09-06 13:40:35 [9,560ms] [Warning] [omni.kit.imgui_renderer.plugin] _createExtendCursor: No windowing.
[10.414s] Simulation App Startup Complete
[10.430s] [ext: omni.kit.streamsdk.plugins-4.5.1] startup
[10.439s] [ext: omni.kit.livestream.core-4.3.6] startup
[10.452s] [ext: omni.kit.livestream.native-4.1.0] startup

Failed to open [/var/run/utmp]
Active user not found. Using default user [kiosk]Streaming server started.
[10.841s] [ext: omni.physx.fabric-106.0.20] startup
[INFO]: Setup complete...

<CLICK CONNECT FROM STREAMING CLIENT ON CLIENT MACHINE>
main: thread_init: already added for thread
main: thread_init: already added for thread
main: thread_init: already added for thread
2024-09-06 13:40:41 [16,423ms] [Warning] [carb.livestream.plugin] onClientEventRaised: NVST_CCE_DISCONNECTED when m_connectionCount 0 != 1

I was able to fix this issue by reverting plugin versions.. see fix section

System Info

Describe the characteristic of your environment: HOST MACHINE

Commit: tag:v1.1.0
Isaac Sim Version: [e.g. 2022.2.0, this can be obtained by cat ${ISAACSIM_PATH}/VERSION]
OS: Ubuntu 20.04
GPU: 2 RTX4090
CUDA: 12.2
GPU Driver: 535.183.01

CLIENT

GPU: TITAN X
OS: ubuntu 20.04
CUDA: 12.2
Driver: 535.183.01
Omniverse launcher: Streaming client 103.1.1

FIX

omni.isaac.lab.app.AppLauncher Line 564: replacing the plugin versions with versions in the working livestream.py example (_isaac_sim/standalone_examples/api/omni.isaac.kit/livestream.py) allows connections.

# Default App: Streaming Client from the Omniverse Launcher
# enable_extension("omni.kit.streamsdk.plugins-4.5.1")
# enable_extension("omni.kit.livestream.core-4.3.6")
# enable_extension("omni.kit.livestream.native-4.1.0")

enable_extension("omni.kit.streamsdk.plugins-3.2.1")
enable_extension("omni.kit.livestream.core-3.2.0")
enable_extension("omni.kit.livestream.native")

Checklist

[X] I have checked that there is no similar issue in the repo (required)
[X] I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

[ ] Isaac lab AppLauncher uses a working streaming configuration
[ ]

alberthli commented 1 week ago

I get this error when trying to stream the headless native app in a docker container.

# start the container using the following command
# see: https://docs.omniverse.nvidia.com/isaacsim/latest/installation/install_container.html#isaac-sim-app-install-container
docker run --name isaac-sim --entrypoint bash -it --runtime=nvidia --gpus all -e "ACCEPT_EULA=Y" --rm --network=host \
    -e "PRIVACY_CONSENT=Y" \
    -v ~/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \
    -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
    -v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \
    -v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
    -v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
    -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
    -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
    -v ~/docker/isaac-sim/documents:/root/Documents:rw \
    nvcr.io/nvidia/isaac-sim:4.1.0

# within the container, start the headless native app
./runheadless.native.sh

The app starts correctly with output:

Loading user config located at: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.1/user.config.json'
[Info] [carb] Logging to file: /root/.nvidia-omniverse/logs/Kit/Isaac-Sim/4.1/kit_20240913_220521.log
2024-09-13 22:05:21 [3ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.1/6a8ac5f2-49e5-4c31-c93e0f88-8c741cc6.dmp.zip'
2024-09-13 22:05:21 [7ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.1/f8fd6a3b-c40d-4629-74075a8f-cf5f7d7e.dmp.zip'
2024-09-13 22:05:21 [9ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.1/be7be3d8-412f-4529-db1aaba4-f66bb6b5.dmp.zip'
2024-09-13 22:05:21 [10ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.1/2ce0020e-02fd-4192-0aee5d8c-eaf1c07f.dmp.zip'
2024-09-13 22:05:21 [25ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.1/8af0efb9-d788-4e4f-4a9b48a3-68d519c9.dmp.zip'
2024-09-13 22:05:21 [34ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.1/30d540e3-2a0c-4bd5-084f639f-e345cb8c.dmp.zip'
[0.534s] [ext: omni.kit.async_engine-0.0.0] startup
[1.342s] [ext: omni.stats-1.0.1] startup
[1.343s] [ext: omni.client-1.1.0] startup
[1.377s] [ext: omni.datastore-0.0.0] startup
[1.377s] [ext: omni.blobkey-1.1.0] startup
[1.378s] [ext: omni.ujitso.default-1.0.0] startup
[1.379s] [ext: omni.hsscclient-0.0.0] startup
[1.380s] [ext: omni.rtx.shadercache.vulkan-1.0.0] startup
[1.381s] [ext: omni.assets.plugins-0.0.0] startup
[1.383s] [ext: omni.gpu_foundation-0.0.0] startup
[1.401s] [ext: carb.windowing.plugins-1.0.0] startup
2024-09-13 22:05:23 [1,372ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-13 22:05:23 [1,372ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[1.402s] [ext: omni.kit.renderer.init-0.0.0] startup
2024-09-13 22:05:23 [1,462ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Server version.
2024-09-13 22:05:24 [2,767ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-13 22:05:24 [2,767ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-09-13 22:05:24 [2,767ms] [Error] [carb.glinterop.plugin] GLInteropContext::init: carb::windowing is not available

|---------------------------------------------------------------------------------------------|
| Driver Version: 550.107.02    | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA H100 NVL                  | Yes: 0 |     | 95830   MB | 10de      | 0          |
|     |                                  |        |     |            | 2321      | 82969629.. |
|     |                                  |        |     |            | 1         |            |
|---------------------------------------------------------------------------------------------|
| 1   | NVIDIA H100 NVL                  | Yes: 1 |     | 95830   MB | 10de      | 0          |
|     |                                  |        |     |            | 2321      | 2a5af47c.. |
|     |                                  |        |     |            | 21        |            |
|---------------------------------------------------------------------------------------------|
| 2   | NVIDIA H100 NVL                  | Yes: 2 |     | 95830   MB | 10de      | 0          |
|     |                                  |        |     |            | 2321      | ba4067c8.. |
|     |                                  |        |     |            | 41        |            |
|---------------------------------------------------------------------------------------------|
| 3   | NVIDIA H100 NVL                  | Yes: 3 |     | 95830   MB | 10de      | 0          |
|     |                                  |        |     |            | 2321      | c97fab96.. |
|     |                                  |        |     |            | 61        |            |
|---------------------------------------------------------------------------------------------|
| 4   | NVIDIA H100 NVL                  | Yes: 4 |     | 95830   MB | 10de      | 0          |
|     |                                  |        |     |            | 2321      | e6c41be8.. |
|     |                                  |        |     |            | 81        |            |
|---------------------------------------------------------------------------------------------|
| 5   | NVIDIA H100 NVL                  | Yes: 5 |     | 95830   MB | 10de      | 0          |
|     |                                  |        |     |            | 2321      | 93fbd922.. |
|     |                                  |        |     |            | a1        |            |
|---------------------------------------------------------------------------------------------|
| 6   | NVIDIA H100 NVL                  | Yes: 6 |     | 95830   MB | 10de      | 0          |
|     |                                  |        |     |            | 2321      | 113faf8e.. |
|     |                                  |        |     |            | c1        |            |
|---------------------------------------------------------------------------------------------|
| 7   | NVIDIA H100 NVL                  | Yes: 7 |     | 95830   MB | 10de      | 0          |
|     |                                  |        |     |            | 2321      | c1f0d0c4.. |
|     |                                  |        |     |            | e1        |            |
|=============================================================================================|
| OS: 22.04.3 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.3, Kernel: 5.15.0-121-generic
| Processor: AMD EPYC 9354 32-Core Processor | Cores: 64 | Logical: 128
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 1547840 | Free Memory: 1531951
| Total Page/Swap (MB): 8191 | Free Page/Swap: 8191
|---------------------------------------------------------------------------------------------|
2024-09-13 22:05:38 [16,450ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 0
2024-09-13 22:05:38 [16,450ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 1
2024-09-13 22:05:38 [16,450ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 2
2024-09-13 22:05:38 [16,450ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 3
2024-09-13 22:05:38 [16,450ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 4
2024-09-13 22:05:38 [16,450ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 5
2024-09-13 22:05:38 [16,450ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 6
2024-09-13 22:05:38 [16,450ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 7
2024-09-13 22:05:38 [16,456ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
[18.411s] [ext: omni.kit.pipapi-0.0.0] startup
[18.417s] [ext: omni.kit.pip_archive-0.0.0] startup
[18.418s] [ext: omni.pip.compute-1.4.0] startup
[18.419s] [ext: omni.pip.cloud-1.1.3] startup
[18.435s] [ext: omni.isaac.core_archive-2.3.0] startup
[18.436s] [ext: omni.isaac.ml_archive-2.0.1] startup
[18.436s] [ext: omni.mtlx-0.1.0] startup
[18.437s] [ext: omni.usd.config-1.0.4] startup
[18.444s] [ext: omni.gpucompute.plugins-0.0.0] startup
[18.444s] [ext: omni.usd.libs-1.0.1] startup
[18.531s] [ext: omni.kit.telemetry-0.5.0] startup
[18.641s] [ext: omni.kit.loop-isaac-1.2.0] startup
[18.644s] [ext: omni.kit.test-0.0.0] startup
[18.933s] [ext: omni.appwindow-1.1.8] startup
2024-09-13 22:05:40 [18,905ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-13 22:05:40 [18,905ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[18.938s] [ext: omni.kit.renderer.core-1.0.1] startup
2024-09-13 22:05:40 [18,926ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-13 22:05:40 [18,926ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-09-13 22:05:40 [18,994ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-13 22:05:40 [18,994ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[19.026s] [ext: omni.kit.renderer.capture-0.0.0] startup
[19.029s] [ext: omni.kit.renderer.imgui-1.0.1] startup
2024-09-13 22:05:40 [19,003ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-13 22:05:40 [19,003ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-09-13 22:05:40 [19,004ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-13 22:05:40 [19,004ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-09-13 22:05:40 [19,005ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-13 22:05:40 [19,005ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[19.150s] [ext: omni.ui-2.23.11] startup
[19.168s] [ext: omni.kit.mainwindow-1.0.3] startup
[19.171s] [ext: carb.audio-0.1.0] startup
[19.196s] [ext: omni.uiaudio-1.0.0] startup
[19.198s] [ext: omni.kit.uiapp-0.0.0] startup
[19.198s] [ext: omni.usd.schema.metrics.assembler-106.0.1] startup
[19.320s] [ext: omni.anim.graph.schema-106.0.2] startup
[19.328s] [ext: omni.usd.schema.audio-0.0.0] startup
[19.348s] [ext: omni.usd.schema.anim-0.0.0] startup
[19.372s] [ext: omni.usd.schema.semantics-0.0.0] startup
[19.377s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[19.384s] [ext: omni.usd.schema.forcefield-106.0.20] startup
[19.411s] [ext: omni.usd.schema.scene.visualization-2.0.2] startup
[19.414s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[19.419s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[19.423s] [ext: omni.activity.core-1.0.1] startup
[19.425s] [ext: omni.usd_resolver-1.0.0] startup
[19.432s] [ext: omni.kit.window.popup_dialog-2.0.24] startup
[19.445s] [ext: omni.resourcemonitor-105.0.1] startup
[19.448s] [ext: omni.kit.actions.core-1.0.0] startup
[19.450s] [ext: omni.usd.core-1.2.11] startup
[19.454s] [ext: omni.kit.widget.nucleus_connector-1.1.8] startup
[19.462s] [ext: omni.graph.exec-0.9.3] startup
[19.463s] [ext: usdrt.scenegraph-7.4.8] startup
[19.513s] [ext: omni.kit.usd_undo-0.1.8] startup
[19.515s] [ext: omni.kit.exec.core-0.13.2] startup
[19.518s] [ext: omni.hydra.usdrt_delegate-7.4.7] startup
[19.538s] [ext: omni.timeline-1.0.10] startup
[19.540s] [ext: omni.kit.commands-1.4.9] startup
[19.552s] [ext: omni.kit.audiodeviceenum-1.0.1] startup
[19.554s] [ext: omni.hydra.scene_delegate-0.3.3] startup
[19.560s] [ext: omni.iray.libs-0.0.0] startup
[19.566s] [ext: omni.usd-1.11.2] startup
[19.661s] [ext: omni.mdl.neuraylib-0.2.5] startup
[19.664s] [ext: omni.kit.asset_converter-2.1.11] startup
2024-09-13 22:05:41 [19,907ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER]   1.1   IRAY   rend warn : CUDA device 0 (NVIDIA H100 NVL): ECC is enabled, this will reduce rendering performance (it is strongly recommended to disable ECC with iray)
2024-09-13 22:05:41 [19,909ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER]   1.1   IRAY   rend warn : CUDA device 1 (NVIDIA H100 NVL): ECC is enabled, this will reduce rendering performance (it is strongly recommended to disable ECC with iray)
2024-09-13 22:05:41 [19,911ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER]   1.1   IRAY   rend warn : CUDA device 2 (NVIDIA H100 NVL): ECC is enabled, this will reduce rendering performance (it is strongly recommended to disable ECC with iray)
2024-09-13 22:05:41 [19,912ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER]   1.1   IRAY   rend warn : CUDA device 3 (NVIDIA H100 NVL): ECC is enabled, this will reduce rendering performance (it is strongly recommended to disable ECC with iray)
2024-09-13 22:05:41 [19,914ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER]   1.1   IRAY   rend warn : CUDA device 4 (NVIDIA H100 NVL): ECC is enabled, this will reduce rendering performance (it is strongly recommended to disable ECC with iray)
2024-09-13 22:05:41 [19,916ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER]   1.1   IRAY   rend warn : CUDA device 5 (NVIDIA H100 NVL): ECC is enabled, this will reduce rendering performance (it is strongly recommended to disable ECC with iray)
2024-09-13 22:05:41 [19,918ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER]   1.1   IRAY   rend warn : CUDA device 6 (NVIDIA H100 NVL): ECC is enabled, this will reduce rendering performance (it is strongly recommended to disable ECC with iray)
2024-09-13 22:05:41 [19,919ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER]   1.1   IRAY   rend warn : CUDA device 7 (NVIDIA H100 NVL): ECC is enabled, this will reduce rendering performance (it is strongly recommended to disable ECC with iray)
[19.969s] [ext: omni.usd.schema.isaac-2.1.0] startup
[19.978s] [ext: omni.usd.schema.physx-106.0.20] startup
[20.006s] [ext: omni.anim.navigation.schema-106.0.2] startup
[20.010s] [ext: omni.ujitso.processor.texture-1.0.0] startup
[20.012s] [ext: omni.volume-0.5.0] startup
[20.015s] [ext: omni.ujitso.client-0.0.0] startup
[20.016s] [ext: omni.kit.clipboard-1.0.3] startup
[20.017s] [ext: omni.kit.collaboration.telemetry-1.0.0] startup
[20.018s] [ext: omni.kit.widget.path_field-2.0.9] startup
[20.023s] [ext: omni.kit.widget.options_menu-1.1.4] startup
[20.034s] [ext: omni.kit.collaboration.channel_manager-1.0.11] startup
[20.039s] [ext: omni.kit.widget.browser_bar-2.0.10] startup
[20.043s] [ext: omni.kit.widget.options_button-1.0.2] startup
[20.045s] [ext: omni.kit.widget.context_menu-1.2.1] startup
[20.051s] [ext: omni.kit.notification_manager-1.0.8] startup
[20.060s] [ext: omni.inspect-1.0.1] startup
[20.062s] [ext: omni.kvdb-106.0.20] startup
[20.065s] [ext: omni.physx.foundation-106.0.20] startup
[20.065s] [ext: omni.localcache-106.0.20] startup
[20.067s] [ext: omni.convexdecomposition-106.0.20] startup
[20.072s] [ext: omni.kit.helper.file_utils-0.1.8] startup
[20.077s] [ext: omni.kit.widget.nucleus_info-1.0.2] startup
[20.079s] [ext: omni.usdphysics-106.0.20] startup
[20.082s] [ext: omni.kit.widget.filebrowser-2.10.48] startup
[20.111s] [ext: omni.kit.search_core-1.0.5] startup
[20.114s] [ext: omni.physx.cooking-106.0.20] startup
[20.126s] [ext: omni.ui.scene-1.9.3] startup
[20.133s] [ext: omni.kit.widget.versioning-1.4.6] startup
[20.143s] [ext: omni.kit.widget.search_delegate-1.0.4] startup
[20.148s] [ext: omni.physx-106.0.20] startup
[20.167s] [ext: omni.kit.window.filepicker-2.10.34] startup
[20.243s] [ext: omni.kit.usd.layers-2.1.31] startup
[20.275s] [ext: omni.kit.menu.utils-1.5.27] startup
[20.317s] [ext: omni.mdl-52.0.1] startup
[20.463s] [ext: omni.kit.window.file_importer-1.1.11] startup
[20.468s] [ext: omni.kit.context_menu-1.8.0] startup
[20.478s] [ext: omni.isaac.version-1.1.0] startup
[20.480s] [ext: omni.physics.tensors-106.0.20] startup
[20.733s] [ext: omni.kit.material.library-1.4.4] startup
[20.758s] [ext: omni.isaac.nucleus-0.3.0] startup
[20.762s] [ext: omni.graph.core-2.170.3] startup
[20.766s] [ext: omni.physx.tensors-106.0.20] startup
[20.793s] [ext: omni.warp.core-1.2.1] startup
[20.990s] [ext: omni.physx.stageupdate-106.0.20] startup
[20.996s] [ext: omni.kit.widget.searchable_combobox-1.0.6] startup
[21.001s] [ext: omni.kit.widget.prompt-1.0.7] startup
[21.005s] [ext: omni.isaac.core-3.18.1] startup
[23.377s] [ext: omni.hydra.rtx-0.2.0] startup
[23.392s] [ext: omni.graph.image.core-0.3.2] startup
[23.397s] [ext: omni.kit.widget.settings-1.1.1] startup
[23.408s] [ext: omni.kit.widget.live_session_management.ui-1.0.1] startup
[23.421s] [ext: omni.graph.tools-1.78.0] startup
[23.546s] [ext: omni.kit.window.preferences-1.5.3] startup
[23.568s] [ext: omni.kit.collaboration.presence_layer-1.0.8] startup
[23.578s] [ext: omni.sensors.tiled-0.0.4] startup
[23.581s] [ext: omni.kit.widget.live_session_management-1.2.18] startup
[23.593s] [ext: omni.hydra.engine.stats-1.0.2] startup
[23.597s] [ext: omni.kit.raycast.query-1.0.5] startup
[23.605s] [ext: omni.kit.viewport.registry-104.0.6] startup
[23.606s] [ext: omni.kit.viewport.legacy_gizmos-1.0.15] startup
[23.610s] [ext: omni.kit.hydra_texture-1.2.6] startup
[23.615s] [ext: omni.kit.window.content_browser_registry-0.0.6] startup
[23.618s] [ext: omni.kit.widget.searchfield-1.1.6] startup
[23.623s] [ext: omni.kit.widget.highlight_label-1.0.2] startup
[23.626s] [ext: omni.kit.widget.viewport-106.0.3] startup
[23.647s] [ext: omni.kit.stage_template.core-1.1.21] startup
[23.649s] [ext: omni.kit.primitive.mesh-1.0.16] startup
[23.666s] [ext: omni.kit.window.file_exporter-1.0.29] startup
[23.669s] [ext: omni.kit.stage_templates-1.2.3] startup
[23.678s] [ext: omni.kit.widget.filter-1.1.4] startup
[23.680s] [ext: omni.kit.hotkeys.core-1.3.3] startup
[23.682s] [ext: omni.kit.window.file-1.3.52] startup
[23.696s] [ext: omni.kit.window.drop_support-1.0.2] startup
[23.698s] [ext: omni.kit.widget.stage-2.10.26] startup
[23.744s] [ext: omni.kit.window.content_browser-2.9.14] startup
[23.773s] [ext: omni.kit.window.property-1.11.1] startup
[23.784s] [ext: omni.kit.viewport.window-106.0.8] startup
[23.835s] [ext: omni.kit.widget.graph-1.12.8] startup
[23.873s] [ext: omni.graph-1.135.0] startup
[24.008s] [ext: omni.kit.property.usd-3.21.28] startup
[24.076s] [ext: omni.kit.viewport.utility-1.0.17] startup
[24.084s] [ext: omni.kit.window.extensions-1.4.9] startup
[24.121s] [ext: omni.kit.widget.text_editor-1.0.2] startup
[24.123s] [ext: omni.kit.widget.material_preview-1.0.16] startup
[24.131s] [ext: omni.ui_query-1.1.2] startup
[24.134s] [ext: omni.graph.action_core-1.1.4] startup
[24.142s] [ext: omni.graph.ui-1.70.0] startup
[24.204s] [ext: omni.kit.ui_test-1.2.18] startup
[24.214s] [ext: omni.kit.manipulator.viewport-107.0.0] startup
[24.218s] [ext: omni.graph.action_nodes-1.23.0] startup
[24.228s] [ext: omni.kit.window.cursor-1.1.2] startup
[24.233s] [ext: omni.graph.visualization.nodes-2.1.1] startup
[24.251s] [ext: omni.graph.action-1.102.1] startup
[24.254s] [ext: omni.kit.viewport.menubar.core-106.0.2] startup
[24.403s] [ext: omni.graph.image.nodes-1.0.2] startup
[24.405s] [ext: omni.isaac.ui-0.16.0] startup
[24.438s] [ext: omni.syntheticdata-0.6.7] startup
[24.532s] [ext: omni.isaac.debug_draw-1.1.0] startup
[24.542s] [ext: omni.kit.viewport.actions-106.0.3] startup
[24.577s] [ext: omni.isaac.range_sensor-3.1.1] startup
[24.665s] [ext: omni.kit.viewport.menubar.display-106.0.2] startup
[24.675s] [ext: omni.debugdraw-0.1.3] startup
[24.681s] [ext: omni.kit.widget.toolbar-1.6.2] startup
[24.704s] [ext: omni.kit.manipulator.transform-104.7.5] startup
[24.715s] [ext: omni.kit.widget.imageview-1.0.3] startup
[24.717s] [ext: omni.usdphysics.ui-106.0.20] startup
[24.747s] [ext: omni.physx.commands-106.0.20] startup
[24.769s] [ext: omni.kit.manipulator.tool.snap-1.4.5] startup
[24.794s] [ext: omni.kit.window.quicksearch-2.4.4] startup
[24.806s] [ext: omni.physx.ui-106.0.20] startup
[24.892s] [ext: omni.kit.property.material-1.9.4] startup
[24.910s] [ext: omni.physx.demos-106.0.20] startup
[25.036s] [ext: omni.kit.property.physx-106.0.20] startup
[25.166s] [ext: omni.kit.manipulator.selector-1.1.1] startup
[25.174s] [ext: omni.physx.vehicle-106.0.20] startup
[25.252s] [ext: omni.kit.viewport.manipulator.transform-106.0.1] startup
[25.259s] [ext: omni.kit.graph.delegate.default-1.2.2] startup
[25.263s] [ext: omni.kit.manipulator.prim.core-107.0.3] startup
[25.292s] [ext: omni.fabric.commands-1.1.4] startup
[25.309s] [ext: omni.kit.graph.editor.core-1.5.3] startup
[25.324s] [ext: omni.kit.manipulator.prim.fabric-106.0.1] startup
[25.329s] [ext: omni.kit.manipulator.prim.usd-106.0.1] startup
[25.335s] [ext: omni.kit.graph.usd.commands-1.3.1] startup
[25.340s] [ext: omni.physx.camera-106.0.20] startup
[25.354s] [ext: omni.physx.cct-106.0.20] startup
[25.374s] [ext: omni.kit.manipulator.selection-104.0.9] startup
[25.385s] [ext: omni.kit.manipulator.prim-106.0.0] startup
[25.387s] [ext: omni.physx.graph-106.0.20] startup
[25.426s] [ext: omni.sensors.nv.common-1.2.2-isaac] startup
[25.444s] [ext: omni.kit.window.toolbar-1.6.1] startup
[25.457s] [ext: omni.kit.widget.layers-1.7.9] startup
[25.532s] [ext: omni.command.usd-1.0.3] startup
[25.538s] [ext: omni.kit.numpy.common-0.1.2] startup
[25.543s] [ext: omni.sensors.nv.wpm-1.2.1-isaac] startup
[25.548s] [ext: omni.physx.supportui-106.0.20] startup
[25.594s] [ext: omni.physx.telemetry-106.0.20] startup
[25.604s] [ext: omni.graph.nodes-1.143.0] startup
Warning: Possible version incompatibility. Attempting to load omni::fabric::IPath with version v0.2 against v0.1.
[25.615s] [ext: omni.graph.scriptnode-1.19.1] startup
[25.621s] [ext: omni.graph.ui_nodes-1.25.1] startup
Warning: Possible version incompatibility. Attempting to load omni::fabric::IPath with version v0.2 against v0.1.
[25.630s] [ext: omni.warp-1.2.1] startup
[25.674s] [ext: omni.physx.bundle-106.0.20] startup
[25.676s] [ext: omni.kit.window.material_graph-1.8.15] startup
[26.166s] [ext: omni.graph.bundle.action-2.0.4] startup
[26.171s] [ext: omni.sensors.nv.materials-1.2.1-isaac] startup
[26.180s] [ext: omni.replicator.core-1.11.14] startup
2024-09-13 22:05:48 [26,525ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
Warp 1.2.1 initialized:
   CUDA Toolkit 11.8, Driver 12.4
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA H100 NVL" (93 GiB, sm_90, mempool enabled)
     "cuda:1"   : "NVIDIA H100 NVL" (93 GiB, sm_90, mempool enabled)
     "cuda:2"   : "NVIDIA H100 NVL" (93 GiB, sm_90, mempool enabled)
     "cuda:3"   : "NVIDIA H100 NVL" (93 GiB, sm_90, mempool enabled)
     "cuda:4"   : "NVIDIA H100 NVL" (93 GiB, sm_90, mempool enabled)
     "cuda:5"   : "NVIDIA H100 NVL" (93 GiB, sm_90, mempool enabled)
     "cuda:6"   : "NVIDIA H100 NVL" (93 GiB, sm_90, mempool enabled)
     "cuda:7"   : "NVIDIA H100 NVL" (93 GiB, sm_90, mempool enabled)
   CUDA peer access:
     Supported fully (all-directional)
   Kernel cache:
     /root/.cache/warp/1.2.1
[27.992s] [ext: omni.isaac.dynamic_control-1.3.8] startup
[28.040s] [ext: omni.sensors.nv.radar-1.2.1-isaac] startup
[28.046s] [ext: omni.sensors.nv.lidar-1.2.2-isaac] startup
[28.051s] [ext: omni.isaac.core_nodes-1.16.1] startup
[28.097s] [ext: omni.isaac.surface_gripper-1.0.1] startup
[28.134s] [ext: omni.isaac.sensor-12.7.1] startup
[28.538s] [ext: omni.isaac.lula-3.0.1] startup
[28.597s] [ext: omni.isaac.motion_generation-7.1.0] startup
[28.882s] [ext: omni.isaac.manipulators-2.1.0] startup
[28.956s] [ext: omni.isaac.quadruped-1.4.5] startup
[29.125s] [ext: omni.isaac.franka-0.4.1] startup
[29.143s] [ext: omni.kit.graph.delegate.modern-1.10.6] startup
[29.186s] [ext: omni.isaac.wheeled_robots-2.3.3] startup
[29.215s] [ext: omni.isaac.universal_robots-0.3.5] startup
[29.230s] [ext: omni.isaac.cortex-0.3.8] startup
[29.242s] [ext: omni.kit.graph.widget.variables-2.1.0] startup
[29.264s] [ext: omni.isaac.cortex.sample_behaviors-1.0.5] startup
[29.270s] [ext: omni.graph.window.core-1.109.0] startup
[29.432s] [ext: omni.isaac.examples-2.2.1] startup
[29.990s] [ext: omni.importer.mjcf-1.1.1] startup
[30.077s] [ext: omni.graph.window.generic-1.24.0] startup
[30.134s] [ext: omni.importer.onshape-0.7.1] startup
[30.160s] [ext: omni.isaac.manipulators.ui-1.2.1] startup
[30.172s] [ext: omni.importer.urdf-1.14.1] startup
[30.268s] [ext: omni.kit.streamsdk.plugins-3.2.1] startup
[30.276s] [ext: omni.isaac.robot_description_editor-2.5.0] startup
[30.368s] [ext: omni.isaac.examples_nodes-0.1.3] startup
[30.382s] [ext: omni.isaac.cloner-0.8.1] startup
[30.402s] [ext: omni.isaac.extension_templates-1.7.0] startup
[30.413s] [ext: omni.isaac.window.about-1.1.0] startup
[30.433s] [ext: omni.isaac.gym-0.11.3] startup
[30.437s] [ext: omni.isaac.import_wizard-0.5.2] startup
[30.463s] [ext: omni.isaac.kit-1.13.0] startup
[30.478s] [ext: omni.anim.curve.core-1.1.13] startup
[30.532s] [ext: omni.isaac.block_world-1.0.0] startup
[30.547s] [ext: omni.kit.preferences.animation-1.1.7] startup
[30.566s] [ext: omni.anim.curve.ui-1.3.16] startup
[30.586s] [ext: omni.isaac.occupancy_map-1.0.2] startup
[30.614s] [ext: omni.anim.shared.core-106.0.0] startup
[30.627s] [ext: omni.isaac.merge_mesh-1.0.2] startup
[30.647s] [ext: omni.anim.curve.bundle-1.2.3] startup
[30.647s] [ext: omni.isaac.occupancy_map.ui-1.1.0] startup
[30.664s] [ext: omni.kit.stage.copypaste-1.2.2] startup
[30.695s] [ext: omni.kit.widget.stage_icons-1.0.4] startup
[30.704s] [ext: omni.kit.widget.zoombar-1.0.5] startup
[30.718s] [ext: omni.kit.usd.collect-2.2.21] startup
[30.742s] [ext: omni.kit.window.stage-2.5.10] startup
[30.757s] [ext: omni.kit.browser.core-2.3.11] startup
[30.792s] [ext: omni.kit.widgets.custom-1.0.8] startup
[30.857s] [ext: omni.kit.tool.collect-2.2.14] startup
[30.879s] [ext: omni.kit.menu.stage-1.2.5] startup
[30.890s] [ext: omni.kit.browser.folder.core-1.9.12] startup
[30.927s] [ext: omni.kit.variant.editor-106.0.0] startup
[31.029s] [ext: omni.kit.core.collection-0.1.7] startup
[31.040s] [ext: omni.isaac.physics_inspector-0.1.4] startup
[31.050s] [ext: omni.isaac.asset_browser-0.5.0] startup
[31.073s] [ext: omni.kit.variant.presenter-105.1.2] startup
[31.095s] [ext: omni.isaac.scene_blox-0.1.2] startup
[31.098s] [ext: omni.isaac.surface_gripper.ui-1.0.0] startup
[31.106s] [ext: omni.isaac.synthetic_recorder-1.8.0] startup
[31.128s] [ext: omni.isaac.throttling-1.1.0] startup
[31.136s] [ext: omni.isaac.utils-1.0.1] startup
[31.148s] [ext: omni.isaac.gain_tuner-1.1.2] startup
[31.166s] [ext: omni.kit.window.usd_paths-1.0.6] startup
[31.182s] [ext: omni.kit.actions.window-1.1.1] startup
[31.205s] [ext: omni.replicator.isaac-1.15.0] startup
[31.323s] [ext: omni.kit.selection-0.1.4] startup
[31.335s] [ext: omni.kit.browser.asset-1.3.9] startup
[31.354s] [ext: omni.kit.hotkeys.window-1.4.5] startup
[31.437s] [ext: omni.kit.menu.aov-1.1.3] startup
[31.453s] [ext: omni.kit.menu.edit-1.1.24] startup
[31.486s] [ext: omni.kit.menu.file-1.1.10] startup
[31.502s] [ext: omni.kit.property.audio-1.0.11] startup
[31.512s] [ext: omni.kit.property.camera-1.0.6] startup
[31.522s] [ext: omni.hydra.scene_api-0.1.2] startup
[31.531s] [ext: omni.kit.property.geometry-1.3.0] startup
[31.548s] [ext: omni.kit.property.light-1.0.8] startup
[31.558s] [ext: omni.kit.property.render-1.1.1] startup
[31.566s] [ext: omni.kit.property.transform-1.5.1] startup
[31.602s] [ext: omni.kit.mesh.raycast-105.4.0] startup
[31.608s] [ext: omni.kit.window.title-1.1.3] startup
2024-09-13 22:05:53 [31,584ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-13 22:05:53 [31,584ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-09-13 22:05:53 [31,589ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-09-13 22:05:53 [31,589ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[31.620s] [ext: omni.isaac.menu-0.5.0] startup
[31.632s] [ext: omni.kit.property.bundle-1.2.11] startup
[31.648s] [ext: omni.kit.property.collection-0.1.17] startup
[31.655s] [ext: omni.kit.menu.create-1.0.13] startup
[31.664s] [ext: omni.kit.property.isaac-0.2.3] startup
[31.675s] [ext: omni.kit.property.layer-1.1.6] startup
[31.692s] [ext: omni.kit.stagerecorder.core-105.0.5] startup
[31.702s] [ext: omni.mdl.usd_converter-1.0.19] startup
[31.733s] [ext: omni.kit.scripting-106.0.1] startup
[31.767s] [ext: omni.kit.search.files-1.0.4] startup
[31.774s] [ext: semantics.schema.property-1.0.3] startup
[31.783s] [ext: omni.kit.stagerecorder.ui-105.0.6] startup
[31.803s] [ext: omni.kit.stage.mdl_converter-1.0.5] startup
[31.811s] [ext: omni.kit.stage_column.payload-2.0.0] startup
[31.820s] [ext: omni.isaac.physics_utilities-0.1.4] startup
[31.830s] [ext: omni.kit.stage_column.variant-1.0.13] startup
[31.848s] [ext: omni.kit.stagerecorder.bundle-105.0.2] startup
[31.849s] [ext: omni.isaac.range_sensor.examples-1.0.0] startup
[31.881s] [ext: omni.graph.window.action-1.26.0] startup
[31.913s] [ext: omni.kit.viewport.menubar.render-106.1.3] startup
[31.939s] [ext: omni.kit.viewport.menubar.settings-106.0.1] startup
[31.970s] [ext: omni.kit.viewport.menubar.camera-105.1.8] startup
[32.005s] [ext: omni.kit.manipulator.camera-105.0.5] startup
[32.035s] [ext: omni.rtx.window.settings-0.6.16] startup
[32.056s] [ext: omni.kit.widget.live-2.1.6] startup
[32.066s] [ext: omni.kit.widget.calendar-1.0.8] startup
[32.079s] [ext: omni.kit.viewport.bundle-104.0.1] startup
[32.080s] [ext: omni.kit.widget.collection-0.1.18] startup
[32.101s] [ext: omni.kit.viewport.menubar.lighting-106.0.2] startup
[32.129s] [ext: omni.kit.viewport.rtx-104.0.1] startup
[32.129s] [ext: omni.kit.viewport_widgets_manager-1.0.8] startup
[32.140s] [ext: omni.kit.widget.cache_indicator-2.0.8] startup
[32.386s] [ext: omni.kit.widget.extended_searchfield-1.0.28] startup
2024-09-13 22:05:54 [32,359ms] [Warning] [omni.kit.widget.cache_indicator.cache_state_menu] Unable to detect Omniverse Cache Server. Consider installing it for better IO performance.
[32.436s] [ext: omni.kit.widget.timeline-105.0.1] startup
[32.460s] [ext: omni.kit.window.collection-0.1.22] startup
[32.469s] [ext: omni.kit.browser.material-1.5.2] startup
[32.493s] [ext: omni.kit.window.commands-0.2.5] startup
[32.506s] [ext: omni.kit.window.console-0.2.12] startup
[32.529s] [ext: omni.kit.thumbnails.usd-1.0.9] startup
[32.867s] [ext: omni.kit.window.privacy-0.1.0] startup
[32.873s] [ext: omni.scene.optimizer.core-106.0.15] startup
[32.967s] [ext: omni.kit.tool.remove_unused.core-0.1.2] startup
[32.973s] [ext: omni.kit.window.material-1.5.7] startup
[33.014s] [ext: omni.kit.window.script_editor-1.7.6] startup
[33.026s] [ext: omni.isaac.wheeled_robots.ui-1.1.1] startup
[33.034s] [ext: omni.kit.tool.asset_exporter-1.3.3] startup
[33.049s] [ext: omni.isaac.range_sensor.ui-1.0.1] startup
[33.057s] [ext: omni.kit.converter.common-500.0.8] startup
[33.128s] [ext: omni.kit.window.status_bar-0.1.6] startup
[33.135s] [ext: omni.kit.tool.remove_unused.controller-0.1.3] startup
[33.143s] [ext: omni.replicator.replicator_yaml-2.0.5] startup
[33.165s] [ext: omni.rtx.settings.core-0.6.0] startup
[33.200s] [ext: omni.usd.metrics.assembler-106.0.1] startup
[33.223s] [ext: omni.kit.converter.dgn_core-500.0.12] startup
[33.306s] [ext: omni.kit.ui.actions-1.0.1] startup
[33.315s] [ext: omni.kit.tool.asset_importer-2.5.5] startup
[33.317s] [ext: omni.kit.converter.hoops_core-500.0.12] startup
[33.330s] [ext: omni.simready.explorer-1.0.26] startup
[33.391s] [ext: omni.kit.converter.jt_core-500.0.12] startup
[33.420s] [ext: omni.usd.metrics.assembler.physics-106.0.2] startup
[33.428s] [ext: omni.usd.metrics.assembler.ui-106.0.2] startup
[33.450s] [ext: omni.kit.menu.common-1.1.5] startup
[33.459s] [ext: omni.kit.converter.dgn-500.0.4] startup
[33.482s] [ext: omni.kit.converter.hoops-500.0.5] startup
[33.491s] [ext: omni.kit.renderer.cuda_interop-1.0.1] startup
[33.497s] [ext: omni.kit.converter.jt-500.0.4] startup
[33.517s] [ext: omni.kit.quicklayout-1.0.7] startup
[33.524s] [ext: semantics.schema.editor-0.3.6] startup
[33.534s] [ext: omni.kit.usda_edit-1.1.11] startup
[33.579s] [ext: omni.isaac.robot_assembler-1.5.0] startup
[33.593s] [ext: omni.isaac.camera_inspector-0.2.2] startup
[33.605s] [ext: omni.kit.livestream.core-3.2.0] startup
[33.611s] [ext: omni.kit.converter.cad-201.1.0-rc.4] startup
[33.612s] [ext: omni.kit.window.stats-0.1.6] startup
[33.617s] [ext: omni.isaac.app.setup-0.13.2] startup
[33.628s] Isaac Sim Headless Native Version: 4.1.0-rc.7
[33.643s] [ext: omni.isaac.assets_check-0.3.4] startup
[33.660s] [ext: omni.kit.profiler.window-2.2.1] startup
2024-09-13 22:05:55 [33,702ms] [Warning] [omni.kit.profiler.window] remove _SpanInstance.__lt__ and use insort 'key' arg instead
[33.746s] [ext: omni.isaac.sim.base-4.1.0] startup
[33.752s] [ext: omni.kit.livestream.native-4.1.0] startup

Failed to open [/var/run/utmp]
Active user not found. Using default user [kiosk]Streaming server started.
[39.193s] [ext: omni.isaac.sim.headless.native-4.1.0] startup
[39.489s] [ext: omni.isaac.ros2_bridge-2.31.4] startup
2024-09-13 22:06:02 [40,118ms] [Warning] [omni.isaac.ros2_bridge.scripts.extension] ROS_DISTRO env var not found, Please source ROS2 Foxy, or Humble, before enabling this extension
[40.148s] Using backup internal ROS2 humble distro
Checking to see if RMW can be loaded:
failed to get symbol 'rmw_init_options_init' due to Environment variable 'AMENT_PREFIX_PATH' is not set or empty, at /workspace/humble_ws/src/rmw_implementation/src/functions.cpp:171, at /workspace/humble_ws/src/rcl/rcl/src/rcl/init_options.c:75
RMW was not loaded

[40.164s] To use the internal libraries included with the extension please set the following environment variables to use with FastDDS (default) or CycloneDDS (ROS2 Humble only): 
RMW_IMPLEMENTATION=rmw_fastrtps_cpp
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/isaac-sim/exts/omni.isaac.ros2_bridge/humble/lib

OR 

RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/isaac-sim/exts/omni.isaac.ros2_bridge/humble/lib
Before starting Isaac Sim
2024-09-13 22:06:02 [40,135ms] [Error] [omni.isaac.ros2_bridge.scripts.extension] ROS2 Bridge startup failed
[75.256s] [ext: omni.isaac.ros2_bridge-2.31.4] shutdown
2024-09-13 22:06:37 [75,385ms] [Warning] [omni.ext._impl._internal] omni.isaac.ros2_bridge-2.31.4 -> <class 'omni.isaac.ros2_bridge.scripts.og_shortcuts.og_shortcut_menu.Extension'>: extension object is still alive, something holds a reference on it. References: ["[0]:type: <class 'method'>, id: 140425045745536", "[1]:type: <class 'method'>, id: 140425092213376", "[2]:type: <class 'method'>, id: 140425045752960", "[3]:type: <class 'method'>, id: 140431943400448", "[4]:type: <class 'method'>, id: 140424885844928", '[*] ...']
[96.474s] app ready
2024-09-13 22:06:58 [96,700ms] [Warning] [omni.kit.imgui_renderer.plugin] _createExtendCursor: No windowing.
2024-09-13 22:06:58 [96,700ms] [Warning] [omni.kit.imgui_renderer.plugin] _createExtendCursor: No windowing.
[96.909s] Isaac Sim Headless Native App is loaded.

When I try to stream to a client computer, a window very briefly pops up and disappears, then on the server, I see

main: thread_init: already added for thread
main: thread_init: already added for thread

When using the -v flag for verbose output, I get a few more lines of error output:

main: thread_init: already added for thread
main: thread_init: already added for thread
2024-09-13 22:28:39 [121,502ms] [Info] [carb.livestream.plugin] Stream server: connected stream 0x7fbb38007a40 on connection 0x7fbaa0012260
2024-09-13 22:28:40 [122,444ms] [Info] [carb.livestream.plugin] Client (nil) disconnected.
2024-09-13 22:28:40 [122,462ms] [Info] [carb.livestream.plugin] Client (nil) disconnected.
2024-09-13 22:28:40 [122,636ms] [Info] [carb.livestream.plugin] Stream Server: stream0 0x7fbaa0012260 (type 1) stopped
2024-09-13 22:28:40 [122,636ms] [Info] [carb.livestream.plugin] Stream Server: stream1 0x7fbaa0007440 (type 4) stopped
2024-09-13 22:28:40 [122,636ms] [Info] [carb.livestream.plugin] Client (nil) disconnected.

On the client, I'm using Omniverse Streaming Client v103.1.1.

Are there any workarounds to this issue?

jtigue-bdai commented 5 days ago

@alberthli, it looks like you are using the fix that came with this issue so I am surprised its not working.

Can you try: settingenable_extension("omni.kit.livestream.native-4.1.0") to enable_extension("omni.kit.livestream.native") in your source/extensions/omni.isaac.lab/omni/isaac/lab/app/app_launcher.py file?

Let me know if that works. If that doesn't work please open another ticket.

alberthli commented 4 days ago

@jtigue-bdai, thanks for the quick response. Sorry if I'm missing something obvious (I'm new to Isaac Lab), but since I'm starting up the server using the script in the Isaac Docker container, I'm not sure I have or am using the app_launcher.py file. I grepd the phrase

omni.kit.livestream.native

in the isaac-sim directory of the container, and it seems like the version for this is just set in a bunch of .kit files. In the file apps/omni.isaac.sim.headless.native.kit, I changed

"omni.kit.livestream.native" = {version = "4.1.0", exact = true}

to

"omni.kit.livestream.native" = {version = "*"}

and ran into the same issue. If I'm doing something wrong, please let me know. Otherwise, I'll open a new issue.

isaac-sim / IsaacLab