jaiveersinghNV
commented
1 year ago Based on the log you've attached, it looks like there were no errors in running the ROS 2 launch graph. The problem may lie in how you are feeding in input messages to the ROS graph.
Could you use the ros2 topic echo tool to check if messages are being published to the input topic /rgb_left?
Also, please check and verify that you have enabled the ROS 2 Bridge in Isaac Sim per the instructions here.
uraninjo
I can easly install and work on Quickstart example but when I try it on Isaac Sim I can't get any output. Firstly, I suspected I didnt install nvidia container but I believe that is not the problem. Not sure...
I also faced the same problem on VSLAM repo.
Quickstart Example: https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_object_detection#quickstart Isaac Sim Example: https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_object_detection/blob/main/docs/tutorial-isaac-sim.md#tutorial-walkthrough
Any suggestions?
[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2023-02-20-10-38-19-647207-openzeka-test-22662 [INFO] [launch]: Default logging verbosity is set to INFO [INFO] [component_container_mt-1]: process started with pid [22673] [INFO] [isaac_ros_detectnet_visualizer.py-2]: process started with pid [22675] [INFO] [rqt_image_view-3]: process started with pid [22677] [component_container_mt-1] [INFO] [1676889500.028942264] [detectnet_container.detectnet_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_dnn_encoders/lib/libdnn_image_encoder_node.so [component_container_mt-1] [INFO] [1676889500.039107248] [NitrosContext]: [NitrosContext] Creating a new shared context [component_container_mt-1] [INFO] [1676889500.039243115] [detectnet_container.detectnet_container]: Found class: rclcpp_components::NodeFactoryTemplate
[component_container_mt-1] [INFO] [1676889500.039274293] [detectnet_container.detectnet_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate
[component_container_mt-1] [INFO] [1676889500.041487616] [dnn_image_encoder]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1676889500.041768563] [dnn_image_encoder]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1676889500.041780112] [dnn_image_encoder]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1676889500.042874622] [dnn_image_encoder]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1676889500.042890917] [dnn_image_encoder]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1676889500.044003013] [dnn_image_encoder]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1676889500.044019705] [dnn_image_encoder]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1676889500.044235413] [dnn_image_encoder]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1676889500.044326936] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/std/libgxf_std.so
[component_container_mt-1] [INFO] [1676889500.046043117] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1676889500.047111142] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1676889500.048479167] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/tensorops/libgxf_tensorops.so
[component_container_mt-1] [INFO] [1676889500.052016290] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/libgxf_message_compositor.so
[component_container_mt-1] [INFO] [1676889500.052427077] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1676889500.052574387] [dnn_image_encoder]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1676889500.055342293] [dnn_image_encoder]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1676889500.166559999] [dnn_image_encoder]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1676889500.168850362] [dnn_image_encoder]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1676889500.175614666] [dnn_image_encoder]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/dnn_image_encoder' in container '/detectnet_container/detectnet_container'
[component_container_mt-1] [INFO] [1676889500.176942643] [detectnet_container.detectnet_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_triton/lib/libisaac_ros_triton_node.so
[component_container_mt-1] [INFO] [1676889500.177786261] [detectnet_container.detectnet_container]: Found class: rclcpp_components::NodeFactoryTemplate
[component_container_mt-1] [INFO] [1676889500.177799764] [detectnet_container.detectnet_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate
[component_container_mt-1] [INFO] [1676889500.179776011] [triton_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1676889500.180062364] [triton_node]: [TritonNode] Set input data format to: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1676889500.180071966] [triton_node]: [TritonNode] Set output data format to: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1676889500.180106136] [triton_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1676889500.180110576] [triton_node]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1676889500.181218905] [triton_node]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1676889500.181235782] [triton_node]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1676889500.182165595] [triton_node]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1676889500.182175651] [triton_node]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1676889500.182290012] [triton_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1676889500.182479427] [triton_node]: [NitrosContext] Loading extension: gxf/triton/libgxf_triton_ext.so
[component_container_mt-1] [INFO] [1676889500.214162780] [triton_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1676889500.215535200] [triton_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1676889500.286901187] [triton_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1676889500.295080111] [triton_node]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1676889500.295516115] [triton_node]: [NitrosPublisherSubscriberGroup] Pinning the component "triton_request/input" (type="nvidia::gxf::DoubleBufferReceiver") to use its compatible format only: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1676889500.297590624] [triton_node]: [NitrosPublisherSubscriberGroup] Pinning the component "vault/vault" (type="nvidia::gxf::Vault") to use its compatible format only: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1676889500.297927094] [triton_node]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/triton_node' in container '/detectnet_container/detectnet_container' [component_container_mt-1] [INFO] [1676889500.299277928] [detectnet_container.detectnet_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_detectnet/lib/libdetectnet_decoder_node.so [component_container_mt-1] [INFO] [1676889500.301595109] [detectnet_container.detectnet_container]: Found class: rclcpp_components::NodeFactoryTemplate
[component_container_mt-1] [INFO] [1676889500.301623949] [detectnet_container.detectnet_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate
[component_container_mt-1] [INFO] [1676889500.303967386] [detectnet_decoder_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1676889500.305013886] [detectnet_decoder_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1676889500.305031625] [detectnet_decoder_node]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1676889500.306238589] [detectnet_decoder_node]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1676889500.306258321] [detectnet_decoder_node]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1676889500.306452578] [detectnet_decoder_node]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1676889500.306461690] [detectnet_decoder_node]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1676889500.306464947] [detectnet_decoder_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1676889500.306712610] [detectnet_decoder_node]: [NitrosContext] Loading extension: gxf/libgxf_detectnet.so
[component_container_mt-1] [INFO] [1676889500.307096282] [detectnet_decoder_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1676889500.307734685] [detectnet_decoder_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1676889500.308943725] [detectnet_decoder_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1676889500.309091762] [detectnet_decoder_node]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1676889500.312390116] [detectnet_decoder_node]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/detectnet_decoder_node' in container '/detectnet_container/detectnet_container' [component_container_mt-1] [INFO] [1676889500.312631831] [triton_node]: Could not negotiate [component_container_mt-1] [INFO] [1676889501.175848196] [dnn_image_encoder]: [NitrosNode] Starting post negotiation setup [component_container_mt-1] [INFO] [1676889501.175903489] [dnn_image_encoder]: [NitrosNode] Getting data format negotiation results [component_container_mt-1] [INFO] [1676889501.175915430] [dnn_image_encoder]: [NitrosPublisher] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32" [component_container_mt-1] [INFO] [1676889501.175922878] [dnn_image_encoder]: [NitrosSubscriber] Negotiation failed [component_container_mt-1] [INFO] [1676889501.175928520] [dnn_image_encoder]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/rgb_left", data_format="nitros_image_bgr8" [component_container_mt-1] [INFO] [1676889501.175961419] [dnn_image_encoder]: [NitrosNode] Exporting the final graph based on the negotiation results [component_container_mt-1] [INFO] [1676889501.184186750] [dnn_image_encoder]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_dnn_encoders/share/isaac_ros_dnn_encoders/KJXJSQBGVX.yaml" [component_container_mt-1] [INFO] [1676889501.184245436] [dnn_image_encoder]: [NitrosNode] Calling user's pre-load-graph callback [component_container_mt-1] [INFO] [1676889501.184697404] [dnn_image_encoder]: In DNN Image Encoder Node preLoadGraphCallback(). [component_container_mt-1] [INFO] [1676889501.184739201] [dnn_image_encoder]: [NitrosNode] Loading application [component_container_mt-1] [INFO] [1676889501.184751907] [dnn_image_encoder]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_dnn_encoders/share/isaac_ros_dnn_encoders/KJXJSQBGVX.yaml' [component_container_mt-1] [INFO] [1676889501.186930498] [dnn_image_encoder]: [NitrosNode] Linking Nitros pub/sub to the loaded application [component_container_mt-1] [INFO] [1676889501.187011516] [dnn_image_encoder]: [NitrosNode] Calling user's post-load-graph callback [component_container_mt-1] [INFO] [1676889501.187019191] [dnn_image_encoder]: In DNN Image Encoder Node postLoadGraphCallback(). [component_container_mt-1] [INFO] [1676889501.187069564] [dnn_image_encoder]: [NitrosContext] Initializing applicaiton... [component_container_mt-1] [INFO] [1676889501.295121372] [dnn_image_encoder]: [NitrosContext] Running appliation... [component_container_mt-1] [INFO] [1676889501.298106752] [triton_node]: [NitrosNode] Starting post negotiation setup [component_container_mt-1] [INFO] [1676889501.298136009] [triton_node]: [NitrosNode] Getting data format negotiation results [component_container_mt-1] [INFO] [1676889501.298144950] [triton_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32" [component_container_mt-1] [INFO] [1676889501.298160580] [triton_node]: [NitrosPublisher] Negotiation failed [component_container_mt-1] [INFO] [1676889501.298167583] [triton_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/tensor_sub", data_format="nitros_tensor_list_nhwc_rgb_f32" [component_container_mt-1] [INFO] [1676889501.298176600] [triton_node]: [NitrosNode] Exporting the final graph based on the negotiation results [component_container_mt-1] [INFO] [1676889501.312571806] [detectnet_decoder_node]: [NitrosNode] Starting post negotiation setup [component_container_mt-1] [INFO] [1676889501.312622392] [detectnet_decoder_node]: [NitrosNode] Getting data format negotiation results [component_container_mt-1] [INFO] [1676889501.312630144] [detectnet_decoder_node]: [NitrosPublisher] Negotiation failed [component_container_mt-1] [INFO] [1676889501.312637144] [detectnet_decoder_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/detectnet/detections", data_format="nitros_detection2_d_array" [component_container_mt-1] [INFO] [1676889501.312645118] [detectnet_decoder_node]: [NitrosSubscriber] Negotiation failed [component_container_mt-1] [INFO] [1676889501.312652408] [detectnet_decoder_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/tensor_sub", data_format="nitros_tensor_list_nchw_rgb_f32" [component_container_mt-1] [INFO] [1676889501.312672077] [detectnet_decoder_node]: [NitrosNode] Exporting the final graph based on the negotiation results [component_container_mt-1] [INFO] [1676889501.313789960] [triton_node]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_triton/share/isaac_ros_triton/HXZGBBNAWY.yaml" [component_container_mt-1] [INFO] [1676889501.313819128] [triton_node]: [NitrosNode] Calling user's pre-load-graph callback [component_container_mt-1] [INFO] [1676889501.313823607] [triton_node]: [NitrosNode] Loading application [component_container_mt-1] [INFO] [1676889501.313828947] [triton_node]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_triton/share/isaac_ros_triton/HXZGBBNAWY.yaml' [component_container_mt-1] [INFO] [1676889501.313997528] [detectnet_decoder_node]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_detectnet/share/isaac_ros_detectnet/EHAGNYUTRS.yaml" [component_container_mt-1] [INFO] [1676889501.314016751] [detectnet_decoder_node]: [NitrosNode] Calling user's pre-load-graph callback [component_container_mt-1] [INFO] [1676889501.314025644] [detectnet_decoder_node]: [NitrosNode] Loading application [component_container_mt-1] [INFO] [1676889501.315185499] [triton_node]: [NitrosNode] Linking Nitros pub/sub to the loaded application [component_container_mt-1] [INFO] [1676889501.315191026] [detectnet_decoder_node]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_detectnet/share/isaac_ros_detectnet/EHAGNYUTRS.yaml' [component_container_mt-1] [INFO] [1676889501.316175337] [detectnet_decoder_node]: [NitrosNode] Linking Nitros pub/sub to the loaded application [component_container_mt-1] [INFO] [1676889501.316221092] [detectnet_decoder_node]: [NitrosNode] Calling user's post-load-graph callback [component_container_mt-1] [INFO] [1676889501.316221089] [triton_node]: [NitrosNode] Calling user's post-load-graph callback [component_container_mt-1] [INFO] [1676889501.316245585] [triton_node]: In TritonNode postLoadGraphCallback(). [component_container_mt-1] [INFO] [1676889501.316354525] [triton_node]: [NitrosContext] Initializing applicaiton... [component_container_mt-1] WARNING: infer_trtis_server.cpp:1212 NvDsTritonServerInit suggest to set model_control_mode:none. otherwise may cause unknow issues. [component_container_mt-1] I0220 10:38:21.375859 22673 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f49ac000000' with size 268435456 [component_container_mt-1] I0220 10:38:21.376048 22673 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864 [component_container_mt-1] I0220 10:38:21.376860 22673 server.cc:563] [component_container_mt-1] +------------------+------+ [component_container_mt-1] | Repository Agent | Path | [component_container_mt-1] +------------------+------+ [component_container_mt-1] +------------------+------+ [component_container_mt-1] [component_container_mt-1] I0220 10:38:21.376875 22673 server.cc:590] [component_container_mt-1] +---------+------+--------+ [component_container_mt-1] | Backend | Path | Config | [component_container_mt-1] +---------+------+--------+ [component_container_mt-1] +---------+------+--------+ [component_container_mt-1] [component_container_mt-1] I0220 10:38:21.376889 22673 server.cc:633] [component_container_mt-1] +-------+---------+--------+ [component_container_mt-1] | Model | Version | Status | [component_container_mt-1] +-------+---------+--------+ [component_container_mt-1] +-------+---------+--------+ [component_container_mt-1] [component_container_mt-1] I0220 10:38:21.389281 22673 metrics.cc:864] Collecting metrics for GPU 0: NVIDIA RTX A4500 [component_container_mt-1] I0220 10:38:21.389441 22673 metrics.cc:757] Collecting CPU metrics [component_container_mt-1] I0220 10:38:21.389527 22673 tritonserver.cc:2264] [component_container_mt-1] +----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [component_container_mt-1] | Option | Value | [component_container_mt-1] +----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [component_container_mt-1] | server_id | triton | [component_container_mt-1] | server_version | 2.26.0 | [component_container_mt-1] | server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace logging | [component_container_mt-1] | model_repository_path[0] | /tmp/models | [component_container_mt-1] | model_control_mode | MODE_EXPLICIT | [component_container_mt-1] | strict_model_config | 1 | [component_container_mt-1] | rate_limit | OFF | [component_container_mt-1] | pinned_memory_pool_byte_size | 268435456 | [component_container_mt-1] | cuda_memory_pool_byte_size{0} | 67108864 | [component_container_mt-1] | response_cache_byte_size | 0 | [component_container_mt-1] | min_supported_compute_capability | 6.0 | [component_container_mt-1] | strict_readiness | 1 | [component_container_mt-1] | exit_timeout | 30 | [component_container_mt-1] +----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [component_container_mt-1] [component_container_mt-1] [INFO] [1676889501.389797883] [triton_node]: [NitrosContext] Running appliation... [component_container_mt-1] [INFO] [1676889501.389999445] [detectnet_decoder_node]: [NitrosContext] Initializing applicaiton... [component_container_mt-1] [INFO] [1676889501.390286804] [detectnet_decoder_node]: [NitrosContext] Running appliation... [component_container_mt-1] I0220 10:38:21.391044 22673 model_lifecycle.cc:459] loading: detectnet:1 [component_container_mt-1] I0220 10:38:21.420699 22673 tensorrt.cc:5442] TRITONBACKEND_Initialize: tensorrt [component_container_mt-1] I0220 10:38:21.420714 22673 tensorrt.cc:5452] Triton TRITONBACKEND API version: 1.10 [component_container_mt-1] I0220 10:38:21.420717 22673 tensorrt.cc:5458] 'tensorrt' TRITONBACKEND API version: 1.10 [component_container_mt-1] I0220 10:38:21.420719 22673 tensorrt.cc:5486] backend configuration: [component_container_mt-1] {"cmdline":{"auto-complete-config":"false","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} [component_container_mt-1] I0220 10:38:21.420981 22673 tensorrt.cc:5591] TRITONBACKEND_ModelInitialize: detectnet (version 1) [component_container_mt-1] I0220 10:38:21.421832 22673 tensorrt.cc:5640] TRITONBACKEND_ModelInstanceInitialize: detectnet (GPU device 0) [component_container_mt-1] I0220 10:38:22.031302 22673 logging.cc:49] Loaded engine size: 22 MiB [component_container_mt-1] I0220 10:38:22.129810 22673 logging.cc:49] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +21, now: CPU 0, GPU 21 (MiB) [component_container_mt-1] I0220 10:38:22.133307 22673 logging.cc:49] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +704, now: CPU 0, GPU 725 (MiB) [component_container_mt-1] W0220 10:38:22.133317 22673 logging.cc:46] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See
CUDA_MODULE_LOADINGin https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars [component_container_mt-1] I0220 10:38:22.133620 22673 tensorrt.cc:1556] Created instance detectnet on GPU 0 with stream priority 0 [component_container_mt-1] I0220 10:38:22.133738 22673 model_lifecycle.cc:693] successfully loaded 'detectnet' version 1