swapnesh-wani-nvidia
commented
1 year ago Could you please provide more info to reproduce the error?
Closed MrOCW closed 1 year ago
swapnesh-wani-nvidia
commented
1 year ago Could you please provide more info to reproduce the error?
7ossamamr
commented
1 year ago As far as I have been experimenting with the ZED2 camera, the published images are encoded in a BGR8 format (you can check this by echoing the image node). However, Nvidia's packages are expecting an RGB8 formatted image. Therefore, you'd need to convert the image to match such encoding and give it a try.
hemalshahNV
commented
1 year ago Could you publish the entire launch graph? The error seems to be coming from the image format converter itself which will output RGB8 for the Apriltag detector but seems to not recognize the input image format and failed. The converter seemed to think it could not support what it thought the input image format was (BGR8 is supported) which indicates either it was not set correctly or set at all. It would be helpful to see what image format is declared on the images coming out from ZED2.
7ossamamr
commented
1 year ago Could you publish the entire launch graph? The error seems to be coming from the image format converter itself which will output RGB8 for the Apriltag detector but seems to not recognize the input image format and failed. The converter seemed to think it could not support what it thought the input image format was (BGR8 is supported) which indicates either it was not set correctly or set at all. It would be helpful to see what image format is declared on the images coming out from ZED2.
I am not so sure who are you actually addressing but since you mentioned the ZED2, I'll consider that question intended to me and take the chance.
I'll start with the output from the isaac_ros_apriltag package while having the zed2 package running while the topics are remapped to suit the requirements of the apriltag detection package.
ros2 launch isaac_ros_apriltag isaac_ros_apriltag.launch.py
[INFO] [launch]: All log files can be found below /root/.ros/log/2022-11-23-09-11-42-938614-sf-robi-433461
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [433486]
[component_container_mt-1] [INFO] [1669194703.416686959] [apriltag_container]: Load Library: /workspaces/orin/install/isaac_ros_apriltag/lib/libapriltag_node.so
[component_container_mt-1] [INFO] [1669194703.511241342] [NitrosContext]: [NitrosContext] Creating a new shared context
[component_container_mt-1] [INFO] [1669194703.511586080] [apriltag_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::apriltag::AprilTagNode>
[component_container_mt-1] [INFO] [1669194703.511652992] [apriltag_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::apriltag::AprilTagNode>
[component_container_mt-1] [INFO] [1669194703.515582228] [apriltag]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1669194703.518840676] [apriltag]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1669194703.518926884] [apriltag]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1669194703.521413072] [apriltag]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1669194703.521525585] [apriltag]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1669194703.522927960] [apriltag]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1669194703.523004312] [apriltag]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1669194703.523619611] [apriltag]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1669194703.525271299] [apriltag]: [NitrosContext] Loading extension: gxf/std/libgxf_std.so
[component_container_mt-1] [INFO] [1669194703.529763513] [apriltag]: [NitrosContext] Loading extension: gxf/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1669194703.531421537] [apriltag]: [NitrosContext] Loading extension: gxf/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1669194703.535110099] [apriltag]: [NitrosContext] Loading extension: gxf/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1669194703.540717391] [apriltag]: [NitrosContext] Loading extension: gxf/libgxf_sight.so
[component_container_mt-1] [INFO] [1669194703.544495745] [apriltag]: [NitrosContext] Loading extension: gxf/libgxf_atlas.so
[component_container_mt-1] [INFO] [1669194703.548871479] [apriltag]: [NitrosContext] Loading extension: gxf/libgxf_isaac_messages.so
[component_container_mt-1] [INFO] [1669194703.552843210] [apriltag]: [NitrosContext] Loading extension: gxf/libgxf_fiducials.so
[component_container_mt-1] [INFO] [1669194703.554912340] [apriltag]: [NitrosContext] Loading extension: gxf/libgxf_message_compositor.so
[component_container_mt-1] [INFO] [1669194703.558207813] [apriltag]: [NitrosContext] Loading extension: gxf/tensorops/libgxf_tensorops.so
[component_container_mt-1] [INFO] [1669194703.579272428] [apriltag]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1669194703.584406245] [apriltag]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1669194703.699002806] [apriltag]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1669194703.703866702] [apriltag]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1669194703.716653357] [apriltag]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/apriltag' in container '/apriltag_container'
[component_container_mt-1] [INFO] [1669194703.802590610] [NitrosContext]: [NitrosContext] Loading application: '/workspaces/dimofac_orin/install/isaac_ros_nitros/share/isaac_ros_nitros/config/type_adapter_nitros_context_graph.yaml'
[component_container_mt-1] [INFO] [1669194703.803220373] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194703.803764088] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194703.923645507] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194703.924977450] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194703.925335468] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194703.926397585] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194703.927197045] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194703.927556438] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194703.928943069] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194703.984645774] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194703.985673555] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194703.987130618] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.075407755] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.075861709] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.077010611] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.139640006] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.141768240] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.142564532] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.207650163] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.209957246] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.210941955] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.277793483] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.279343762] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.279957237] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.340417950] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.343985807] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.344723955] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.408191850] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.410653078] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.411331289] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.473096648] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.474697808] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.475209458] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.540632947] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.542298523] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.543476673] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.605328720] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.607264601] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.607897244] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.672918555] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.674966981] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.675844777] [apriltag]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1669194704.717221620] [apriltag]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1669194704.717331413] [apriltag]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1669194704.717360085] [apriltag]: [NitrosPublisher] Negotiation failed
[component_container_mt-1] [INFO] [1669194704.717378101] [apriltag]: [NitrosPublisher] Use only the compatible publisher: topic_name="/tag_detections", data_format="nitros_april_tag_detection_array"
[component_container_mt-1] [INFO] [1669194704.717397717] [apriltag]: [NitrosSubscriber] Negotiation failed
[component_container_mt-1] [INFO] [1669194704.717410517] [apriltag]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1669194704.717423861] [apriltag]: [NitrosSubscriber] Negotiation failed
[component_container_mt-1] [INFO] [1669194704.717434933] [apriltag]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/image", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1669194704.717526294] [apriltag]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1669194704.729404336] [apriltag]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/dimofac_orin/install/isaac_ros_apriltag/share/isaac_ros_apriltag/RDSXMYEKWZ.yaml"
[component_container_mt-1] [INFO] [1669194704.729518896] [apriltag]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1669194704.729535376] [apriltag]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1669194704.729550129] [apriltag]: [NitrosContext] Loading application: '/workspaces/dimofac_orin/install/isaac_ros_apriltag/share/isaac_ros_apriltag/RDSXMYEKWZ.yaml'
[component_container_mt-1] [INFO] [1669194704.732973185] [apriltag]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1669194704.733153282] [apriltag]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1669194704.733239811] [apriltag]: [NitrosContext] Initializing applicaiton...
[component_container_mt-1] [INFO] [1669194704.853501264] [apriltag]: [NitrosContext] Running appliation...
[component_container_mt-1] 2022-11-23 09:11:44.856 ERROR extensions/tensor_ops/detail/ImageAdapterVideoBufferImpl.cpp@61: invalid video format.
[component_container_mt-1] 2022-11-23 09:11:44.856 WARN gxf/std/greedy_scheduler.cpp@221: Error while executing entity 188 named 'RDSXMYEKWZ_image_converter': GXF_FAILURE
[component_container_mt-1] 2022-11-23 09:11:44.858 ERROR external/com_nvidia_gxf/gxf/std/entity_executor.cpp@200: Entity with 211 not found!
[component_container_mt-1] [ERROR] [1669194704.858253095] [apriltag]: [NitrosPublisher] Vault ("vault/vault", eid=211) was stopped. The graph may have been terminated due to an error.
[component_container_mt-1] terminate called after throwing an instance of 'std::runtime_error'
[component_container_mt-1] what(): [NitrosPublisher] Vault ("vault/vault", eid=211) was stopped. The graph may have been terminated due to an error.
[ERROR] [component_container_mt-1]: process has died [pid 433486, exit code -6, cmd '/opt/ros/humble/install/lib/rclcpp_components/component_container_mt --ros-args -r __node:=apriltag_container -r __ns:=/'].For the output images from the ZED2 they are bgra8 encoded.
header:
stamp:
sec: 1669194657
nanosec: 274156592
frame_id: zed2_left_camera_optical_frame
height: 360
width: 640
encoding: bgra8
is_bigendian: 0
step: 2560
data:
- 91
- 128
- 177
- 255
- 93
- 130
- 178
- 255
- 94
- 130
- 178
- 255
- 96
- 133
- 181
- 255
- 95
- 132
- 180
- 255
- 94
- 131
- 179
- 255
- 97
- 133
- 181
- 255
- 97
- 134
- 181
- 255
- 97
- 133
- 184
- 255
- 99
- 135
- 185
- 255
- 101
- 137
- 187
- 255
- 99
- 135
- 185
- 255
- 99
- 135
- 185
- 255
- 100
- 136
- 186
- 255
- 102
- 138
- 188
- 255
- 102
- 138
- 188
- 255
- 100
- 136
- 186
- 255
- 103
- 138
- 185
- 255
- 102
- 137
- 185
- 255
- 101
- 137
- 185
- 255
- 105
- 139
- 188
- 255
- 105
- 139
- 187
- 255
- 105
- 138
- 186
- 255
- 106
- 138
- 186
- 255
- 104
- 138
- 186
- 255
- 107
- 140
- 188
- 255
- 107
- 139
- 188
- 255
- 107
- 139
- 188
- 255
- 105
- 138
- 186
- 255
- 104
- 137
- 185
- 255
- 106
- 139
- 187
- 255
- 103
- 135
- 184
- 255
- '...'However, when I converted the bgr8 to rgb8 and had the apriltag detector subscribed to the published converted stream, it worked fine.
swapnesh-wani-nvidia
commented
1 year ago Zed2 is producing images encoded as bgra8 (4 channel) which is not supported. You can either change it to bgr8 or rgb8 (3 channels) and then pass it to the rest of the graph.
7ossamamr
commented
1 year ago Yeah exactly, that was what I mentioned earlier to @MrOCW regarding his question as this is what worked for me.
Thank you for the follow up.
MrOCW
commented
1 year ago Thanks alot for the answers! haven't had time to work on this but I'll trust that this solution will work out :)
Running on a Jetson Orin AGX JP 5.0.2