Closed BADAL244 closed 1 year ago
jaiveersinghNV
commented
1 year ago Could you clear your build/ and install/ directories, rebuild the workspace, and try again? The libtritonserver.so file is built when the isaac_ros_triton package in the isaac_ros_dnn_inference repository is built using colcon build.
BADAL244
commented
1 year ago libtritonserver.so is this a part of DeepStream ? DO i have to install DeepStream to solve this .removing the build and install doesn't make sense. I am able to run tensorrt example , triton showing this error.
swapnesh-wani-nvidia
commented
1 year ago libtritonserver.so is part of isaac_ros_triton package. You don't have to do anything with DeepStream.
HappySamuel
commented
7 months ago Hi @jaiveersinghNV , @swapnesh-wani-nvidia
Could you clear your
build/andinstall/directories, rebuild the workspace, and try again? Thelibtritonserver.sofile is built when theisaac_ros_tritonpackage in theisaac_ros_dnn_inferencerepository is built usingcolcon build.
I have ensure that isaac_ros_triton package in the isaac_ros_dnn_inference, and i did many times of clear build & install directories, rebuild the workspace using colcon build. But none of them successfully generated libtritonserver.so. Any idea why? and any solution?
Best Samuel
HappySamuel
commented
7 months ago Hi @BADAL244
Have you resolve this libtritonserver.so missing error? I tried a lot of times on rebuilding workspace using colcon build. But the libtritonserver.so still never appear.
Best, Samuel
ros2 launch isaac_ros_unet isaac_ros_unet_triton.launch.py model_name:=peoplesemsegnet_shuffleseg model_repository_paths:=['/home/badal/data/tao-converter_vv3.22.05_trt8.4_aarch64/models'] input_binding_names:=['input_2:0'] output_binding_names:=['argmax_1'] network_output_type:='argmax' [INFO] [launch]: All log files can be found below /home/badal/.ros/log/2023-07-25-11-56-08-642323-badal-desktop-7416 [INFO] [launch]: Default logging verbosity is set to INFO [INFO] [component_container_mt-1]: process started with pid [7430] [component_container_mt-1] [INFO] [1690257371.992358167] [unet_container]: Load Library: /home/badal/vslam_ws/install/isaac_ros_dnn_encoders/lib/libdnn_image_encoder_node.so [component_container_mt-1] [INFO] [1690257372.613882499] [unet_container]: Found class: rclcpp_components::NodeFactoryTemplate
[component_container_mt-1] [INFO] [1690257372.615917419] [unet_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate
[component_container_mt-1] [INFO] [1690257372.746378537] [NitrosContext]: [NitrosContext] Creating a new shared context
[component_container_mt-1] [INFO] [1690257372.748530743] [dnn_image_encoder]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1690257372.774453896] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1690257372.820423740] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_gxf_helpers.so
[component_container_mt-1] [INFO] [1690257372.853413686] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_sight.so
[component_container_mt-1] [INFO] [1690257372.890557894] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_atlas.so
[component_container_mt-1] [INFO] [1690257372.924691291] [NitrosContext]: [NitrosContext] Loading application: '/home/badal/vslam_ws/install/isaac_ros_nitros/share/isaac_ros_nitros/config/type_adapter_nitros_context_graph.yaml'
[component_container_mt-1] [INFO] [1690257372.936783271] [NitrosContext]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1690257372.959984396] [NitrosContext]: [NitrosContext] Running application...
[component_container_mt-1] 2023-07-25 11:56:12.960 WARN gxf/std/program.cpp@456: No system specified. Nothing to do
[component_container_mt-1] [INFO] [1690257373.001809899] [dnn_image_encoder]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1690257373.002648982] [dnn_image_encoder]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1690257373.038191794] [dnn_image_encoder]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1690257373.039342413] [dnn_image_encoder]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1690257373.067547858] [dnn_image_encoder]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1690257373.069476244] [dnn_image_encoder]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1690257373.073576422] [dnn_image_encoder]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1690257373.088115023] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/libgxf_message_compositor.so
[component_container_mt-1] [INFO] [1690257373.122406763] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1690257373.156205133] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1690257373.194179654] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_tensorops.so
[component_container_mt-1] [INFO] [1690257373.297673434] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1690257373.317155839] [dnn_image_encoder]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1690257373.396755307] [dnn_image_encoder]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1690257375.103648168] [dnn_image_encoder]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1690257375.148956853] [dnn_image_encoder]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1690257375.375138305] [dnn_image_encoder]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/dnn_image_encoder' in container '/unet_container'
[component_container_mt-1] [INFO] [1690257375.447019996] [unet_container]: Load Library: /home/badal/vslam_ws/install/isaac_ros_triton/lib/libtriton_node.so
[component_container_mt-1] [INFO] [1690257375.462966539] [unet_container]: Found class: rclcpp_components::NodeFactoryTemplate
[component_container_mt-1] [INFO] [1690257375.463921084] [unet_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate
[component_container_mt-1] [INFO] [1690257375.576637439] [triton_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1690257375.617393603] [triton_node]: [TritonNode] Set input data format to: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1690257375.618090310] [triton_node]: [TritonNode] Set output data format to: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1690257375.618979604] [triton_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1690257375.619177502] [triton_node]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1690257375.640717323] [triton_node]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1690257375.642276891] [triton_node]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1690257375.655154765] [triton_node]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1690257375.656179010] [triton_node]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1690257375.659424423] [triton_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1690257375.713413680] [triton_node]: [NitrosContext] Loading extension: gxf/triton/libgxf_triton_ext.so
[component_container_mt-1] [ERROR] [1690257375.741904960] [triton_node]: [NitrosContext] dlopen failed when opening "/home/badal/vslam_ws/install/isaac_ros_triton/share/isaac_ros_triton/gxf/triton/libgxf_triton_ext.so": libtritonserver.so: cannot open shared object file: No such file or directory
[component_container_mt-1] [ERROR] [1690257375.743142240] [triton_node]: [NitrosNode] loadExtensions Error: GXF_FAILURE
[component_container_mt-1] [INFO] [1690257375.744931227] [triton_node]: [NitrosNode] Terminating the running application
[component_container_mt-1] [INFO] [1690257375.745580508] [triton_node]: [NitrosContext] Interrupting GXF...
[component_container_mt-1] 2023-07-25 11:56:15.745 ERROR gxf/std/program.cpp@475: Attempted interrupting when not running (state=0d).
[component_container_mt-1] 2023-07-25 11:56:15.746 ERROR gxf/core/runtime.cpp@1243: Graph interrupt failed with error: GXF_INVALID_EXECUTION_SEQUENCE
[component_container_mt-1] [ERROR] [1690257375.746065973] [triton_node]: [NitrosContext] GxfGraphInterrupt Error: GXF_INVALID_EXECUTION_SEQUENCE
[component_container_mt-1] [INFO] [1690257375.746224029] [triton_node]: [NitrosContext] Waiting on GXF...
[component_container_mt-1] [INFO] [1690257375.746382885] [triton_node]: [NitrosContext] Deinitializing...
[component_container_mt-1] [INFO] [1690257375.746627218] [triton_node]: [NitrosContext] Destroying context
[component_container_mt-1] [INFO] [1690257375.746925377] [triton_node]: [NitrosNode] Application termination done
[component_container_mt-1] [ERROR] [1690257375.813321796] [unet_container]: Component constructor threw an exception: [NitrosNode] loadExtensions Error: GXF_FAILURE
[ERROR] [launch_ros.actions.load_composable_nodes]: Failed to load node 'triton_node' of type 'nvidia::isaac_ros::dnn_inference::TritonNode' in container '/unet_container': Component constructor threw an exception: [NitrosNode] loadExtensions Error: GXF_FAILURE
[component_container_mt-1] [INFO] [1690257375.876381788] [unet_container]: Load Library: /home/badal/vslam_ws/install/isaac_ros_unet/lib/libunet_decoder_node.so
[component_container_mt-1] [INFO] [1690257375.906204881] [unet_container]: Found class: rclcpp_components::NodeFactoryTemplate
[component_container_mt-1] [INFO] [1690257375.906834577] [unet_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate
[component_container_mt-1] [INFO] [1690257375.971981428] [unet_decoder_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1690257375.996895277] [unet_decoder_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1690257375.997465259] [unet_decoder_node]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1690257376.016359824] [unet_decoder_node]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1690257376.016795590] [unet_decoder_node]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1690257376.022548236] [unet_decoder_node]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1690257376.022906879] [unet_decoder_node]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1690257376.023009348] [unet_decoder_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1690257376.069812699] [unet_decoder_node]: [NitrosContext] Loading extension: gxf/libgxf_segmentation_postprocessor.so
[component_container_mt-1] [INFO] [1690257376.104115859] [unet_decoder_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1690257376.135671487] [unet_decoder_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1690257376.384891003] [dnn_image_encoder]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1690257376.385379764] [dnn_image_encoder]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1690257376.385521179] [dnn_image_encoder]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1690257376.385641602] [dnn_image_encoder]: [NitrosPublisher] Use only the compatible publisher: topic_name="/tensor_pub", data_format="nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1690257376.385747047] [dnn_image_encoder]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1690257376.385877806] [dnn_image_encoder]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/image", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1690257376.386190014] [dnn_image_encoder]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1690257376.400752166] [unet_decoder_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1690257376.422335156] [unet_decoder_node]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1690257376.423596533] [unet_decoder_node]: [NitrosPublisherSubscriberGroup] Pinning the component "colored_segmentation_mask_sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1690257376.458283969] [unet_decoder_node]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/unet_decoder_node' in container '/unet_container'
[component_container_mt-1] [INFO] [1690257376.486819411] [dnn_image_encoder]: [NitrosNode] Wrote the final top level YAML graph to "/home/badal/vslam_ws/install/isaac_ros_dnn_encoders/share/isaac_ros_dnn_encoders/PVHACSQUUQ.yaml"
[component_container_mt-1] [INFO] [1690257376.487199910] [dnn_image_encoder]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1690257376.487320876] [dnn_image_encoder]: In DNN Image Encoder Node preLoadGraphCallback().
[component_container_mt-1] [INFO] [1690257376.487569785] [dnn_image_encoder]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1690257376.487682943] [dnn_image_encoder]: [NitrosContext] Loading application: '/home/badal/vslam_ws/install/isaac_ros_dnn_encoders/share/isaac_ros_dnn_encoders/PVHACSQUUQ.yaml'
[component_container_mt-1] [INFO] [1690257376.522047706] [dnn_image_encoder]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1690257376.522861988] [dnn_image_encoder]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1690257376.523011115] [dnn_image_encoder]: In DNN Image Encoder Node postLoadGraphCallback().
[component_container_mt-1] [INFO] [1690257376.523599178] [dnn_image_encoder]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1690257376.808943355] [dnn_image_encoder]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1690257376.809556122] [dnn_image_encoder]: [NitrosNode] Starting a heartbeat timer (eid=22)
[component_container_mt-1] [INFO] [1690257377.463722196] [unet_decoder_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1690257377.464016675] [unet_decoder_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1690257377.464126088] [unet_decoder_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1690257377.464262735] [unet_decoder_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/unet/raw_segmentation_mask", data_format="nitros_image_mono8"
[component_container_mt-1] [INFO] [1690257377.464354260] [unet_decoder_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1690257377.464478938] [unet_decoder_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/unet/colored_segmentation_mask", data_format="nitros_image_rgb8"
[component_container_mt-1] [INFO] [1690257377.464603553] [unet_decoder_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1690257377.464713510] [unet_decoder_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/tensor_sub", data_format="nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1690257377.464947346] [unet_decoder_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1690257377.504186117] [unet_decoder_node]: [NitrosNode] Wrote the final top level YAML graph to "/home/badal/vslam_ws/install/isaac_ros_unet/share/isaac_ros_unet/ARALKDMKSC.yaml"
[component_container_mt-1] [INFO] [1690257377.504499797] [unet_decoder_node]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1690257377.504596186] [unet_decoder_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1690257377.504739361] [unet_decoder_node]: [NitrosContext] Loading application: '/home/badal/vslam_ws/install/isaac_ros_unet/share/isaac_ros_unet/ARALKDMKSC.yaml'
[component_container_mt-1] [INFO] [1690257377.516895822] [unet_decoder_node]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1690257377.517669109] [unet_decoder_node]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1690257377.518208465] [unet_decoder_node]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1690257377.530851894] [unet_decoder_node]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1690257377.532258334] [unet_decoder_node]: [NitrosNode] Starting a heartbeat timer (eid=101)
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[component_container_mt-1] [INFO] [1690257482.522108454] [unet_decoder_node]: [NitrosNode] Terminating the running application [component_container_mt-1] [INFO] [1690257482.522230859] [unet_decoder_node]: [NitrosContext] Interrupting GXF... [component_container_mt-1] [INFO] [1690257482.524954285] [unet_decoder_node]: [NitrosContext] Waiting on GXF... [component_container_mt-1] [INFO] [1690257482.525231258] [unet_decoder_node]: [NitrosContext] Deinitializing... [component_container_mt-1] [INFO] [1690257482.535484963] [unet_decoder_node]: [NitrosContext] Destroying context [component_container_mt-1] [INFO] [1690257482.536794497] [unet_decoder_node]: [NitrosNode] Application termination done [component_container_mt-1] [INFO] [1690257482.596890640] [dnn_image_encoder]: [NitrosNode] Terminating the running application [component_container_mt-1] [INFO] [1690257482.597173246] [dnn_image_encoder]: [NitrosContext] Interrupting GXF... [component_container_mt-1] [INFO] [1690257482.597257058] [dnn_image_encoder]: [NitrosContext] Waiting on GXF... [component_container_mt-1] [INFO] [1690257482.597832989] [dnn_image_encoder]: [NitrosContext] Deinitializing... [component_container_mt-1] [INFO] [1690257482.657636575] [dnn_image_encoder]: [NitrosContext] Destroying context [component_container_mt-1] [INFO] [1690257482.663163718] [dnn_image_encoder]: [NitrosNode] Application termination done [component_container_mt-1] |==================================================================================================================================================================| [component_container_mt-1] | Job Statistics Report (regular) | [component_container_mt-1] |==================================================================================================================================================================| [component_container_mt-1] | Name | Count | Time (Median - 90% - Max) [ms] | Load (%) | Exec(ms) | Variation (Median - 90% - Max) [ns] | [component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------| [component_container_mt-1] |==================================================================================================================================================================| [component_container_mt-1] |==================================================================================================================================================================| [component_container_mt-1] | Entity Statistics Report (regular) | [component_container_mt-1] |==================================================================================================================================================================| [component_container_mt-1] | Entity Name | Entity State | Count | Time (Median - 90% - Max) [ms] | [component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------| [component_container_mt-1] |==================================================================================================================================================================| [INFO] [component_container_mt-1]: process has finished cleanly [pid 7430]