search

NVIDIA-ISAAC-ROS / isaac_ros_image_pipeline

NVIDIA-accelerated ROS2 packages for camera image processing.
https://developer.nvidia.com/isaac-ros-gems
Apache License 2.0
112 stars 19 forks source link

GXF_FAILURE, tensor stream creation failed. Jetson AGX Orin and AGX Xavier (JP 5.1.1) #38

Closed raitzjm closed 7 months ago

raitzjm commented 9 months ago

When I try to launch an Isaac node on my Jetson that creates a tensor stream using isaac_ros_image_pipeline/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/TensorStrea m.cpp, I get the following error (same as https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_compression/issues/2)

[component_container_mt-2] 2023-09-15 14:33:11.362 ERROR /workspaces/isaac_ros-dev/src/isaac_ros_image_pipeline/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/TensorStrea m.cpp@104: tensor stream creation failed.
[component_container_mt-2] 2023-09-15 14:33:11.363 ERROR gxf/std/entity_warden.cpp@380: Failed to initialize component 00292 (stream) [component_container_mt-2] 2023-09-15 14:33:11.363 ERROR gxf/core/runtime.cpp@616: Could not initialize entity 'IPOQAFVQYV_global' (E290): GXF_FAILURE [component_container_mt-2] 2023-09-15 14:33:11.363 ERROR gxf/std/program.cpp@205: Failed to activate entity 00290 named IPOQAFVQYV_global: GXF_FAILURE [component_container_mt-2] 2023-09-15 14:33:11.363 ERROR gxf/std/program.cpp@207: Deactivating... [component_container_mt-2] 2023-09-15 14:33:11.396 ERROR gxf/core/runtime.cpp@1227: Graph activation failed with error: GXF_FAILURE [component_container_mt-2] [ERROR] [1694759591.396604150] [left_decoder_node]: [NitrosContext] GxfGraphActivate Error: GXF_FAILURE [component_container_mt-2] [INFO] [1694759591.396906843] [right_decoder_node]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_h264_decoder/share/isaacros h264_decoder/EZXPATGEMM.yaml'
[component_container_mt-2] [ERROR] [1694759591.396922620] [left_decoder_node]: [NitrosNode] runGraphAsync Error: GXF_FAILURE [component_container_mt-2] terminate called after throwing an instance of 'std::runtime_error' [component_container_mt-2] what(): [NitrosNode] runGraphAsync Error: GXF_FAILURE

I tried launching the h264 encoder and decoder from the isaac_ros_compression package and the ResizeNode and ImageFormatConverterNode from isaac_ros_image_proc . All of them gave me the same error, I tested it with a Jetson AGX Orin dev-kit, a AGX Xavier dev-kit, and another AGX Orin System from a 3rd party vendor, all of them running JetPack 5.1.1. I tried the h264 decoder node in an x86 sytem as well and it worked there.

swapnesh-wani-nvidia commented 8 months ago

Hello, could you explain in more detail about your system configuration? Are you accessing Jetson AGX Orin over SSH with X11 forwarding? Try upgrading your Jetpack version to 5.1.2 and use the latest release of the Isaac ROS https://nvidia-isaac-ros.github.io/releases/index.html#isaac-ros-2-0-0-october-18-2023.

raitzjm commented 8 months ago

I was accessing the Jetson over SSH with X11 forwarding enable, but I did not run any GUI application. I updated my Jetson to Jetpack 5.1.2 and Isaac 2.0.0 and tried again, but I am still having the same issue regardless.

swapnesh-wani-nvidia commented 8 months ago

There are additional steps that you need to do in order to tun GPU-based applications on a remote machine with X11 forwarding enabled. See this post. You can quickly try to disable the X11 forwarding and check if the problem is solved. If yes, then the problem is distilled to X11 forwarding.