Intermediate image in RQT is not visible for Perception Demo

[jasvinderkhurana]

I have ran Perception Graph application referring to this page. I am able to see the compute graph using RQT application, but When I try to see intermediate images using RQT I just see a gradient image. Below are the steps I used to execute application:

On x86 machine:

• Run Gazebo

cd ~/krs_ws
source /opt/ros/humble/setup.bash 
colcon build --merge-install
source install/setup.bash
ros2 launch perception_2nodes simulation.launch.py

#after this step you should be able to see the Gazebo application simulating the camera setup.

• On Seperate Terminal execute RQT

  source /opt/ros/humble/setup.bash
  rqt

• In RQT application open, Plugins→ Introspection→ Node Graph
• In RQT application open Plugins → Visualization → Image View

On KR260 Board

• Run Below Commands:

  source /opt/ros/humble/setup.bash
  cd krs_ws/
  source install-kr260-ubuntu/local_setup.bash
  ros2 launch perception_2nodes trace_rectify_resize_fpga.launch.py

Observe the behaviour on RQT, intermediate images are not shown

[jasvinderkhurana]

@vmayoral , can you please help to understand what would be the possible issue here.

[vmayoral]

Thanks for reporting this @jasvinderkhurana, I'll try and have a look at this next week.

[vmayoral]

All right, so I looked at the robotics perception acceleration example. First validated interoperability locally. Then tested things with embedded targets. Note that from your description above, this is an interoperability issue, not a functionality one so I'm leaving aside acceleration kernels to reduce complexity and focusing on reproducing the interoperability issues you describe. Here're my thoughts and conclusions:

✅ Robotics perception: using local amd64 workstation

Using Vitis 2022.1.

# launch simulation (skipping intermediate steps which might be necessary depending on your setup)
source /opt/ros/humble/setup.bash
colcon build --merge-install
source install/setup.bash
ros2 launch perception_2nodes simulation.launch.py

# launch a simple graph
source /opt/ros/humble/setup.bash
source install/setup.bash
ros2 launch perception_2nodes trace_rectify_resize.launch.py

This works just fine.

✅ Robotics perception: using KV260 accelerator

Using Vitis 2021.2 and acceleration_firmware_kv260 v0.9.0 (Note this is leveraging ROS Rolling in the accelerator, and interoperating with ROS Humble in the workstation):

Prepare the accelerator

source /opt/ros/humble/setup.bash
colcon build --merge-install
source install/setup.bash
colcon acceleration select kv260
# cross-compile ROS 2 packages
colcon build --build-base=build-kv260 --install-base=install-kv260 --merge-install --mixin kv260 --cmake-args -DNOKERNELS=true  
# then build accelerators, as necessary (not needed in this case, just testing interoperability)
# (...)
# create a raw disk image for the SD card with PetaLinux’s rootfs, then burn it into a micro SD card
colcon acceleration linux vanilla --install-dir install-kv260

Launch stuff and test things with KV260 accelerator

Workstation:

# launch simulation (skipping intermediate steps which might be necessary depending on your setup)
source /opt/ros/humble/setup.bash
colcon build --merge-install
source install/setup.bash
ros2 launch perception_2nodes simulation.launch.py

Accelerator (KV260):

source /opt/ros/rolling/setup.bash
source ros2_ws/local_setup.bash
ros2 launch perception_2nodes trace_rectify_resize.launch.py

This works just fine as well.

✅ Robotics perception: using KV260 accelerator with Ubuntu 22.04 image

Built ROS 2 workspace in the accelerator (native build)

# create workspace and fetch sources, except firmware, which isn't needed if we're not cross-compiling
mkdir -p ~/krs_ws/src; cd ~/krs_ws
cat << 'EOF' > krs_humble.repos
repositories:  
  perception/image_pipeline:
    type: git
    url: https://github.com/ros-acceleration/image_pipeline
    version: ros2

  tracing/tracetools_acceleration:
    type: git
    url: https://github.com/ros-acceleration/tracetools_acceleration
    version: humble

  acceleration/adaptive_component:
    type: git
    url: https://github.com/ros-acceleration/adaptive_component
    version: humble
  acceleration/ament_acceleration:
    type: git
    url: https://github.com/ros-acceleration/ament_acceleration
    version: humble
  acceleration/ament_vitis:
    type: git
    url: https://github.com/ros-acceleration/ament_vitis
    version: humble
  acceleration/colcon-hardware-acceleration:
    type: git
    url: https://github.com/colcon/colcon-hardware-acceleration
    version: main
  acceleration/ros2_kria:
    type: git
    url: https://github.com/ros-acceleration/ros2_kria
    version: main
  acceleration/ros2acceleration:
    type: git
    url: https://github.com/ros-acceleration/ros2acceleration
    version: humble
  acceleration/vitis_common:
    type: git
    url: https://github.com/ros-acceleration/vitis_common
    version: humble
  acceleration/acceleration_examples:
    type: git
    url: https://github.com/ros-acceleration/acceleration_examples
    version: main
EOF
vcs import src --recursive < krs_humble.repos

# fetch all dependencies of source code
sudo rosdep init
rosdep update
rosdep install --from-paths src --ignore-src --rosdistro humble -y --skip-keys "console_bridge fastcdr fastrtps libopensplice67 libopensplice69 rti-connext-dds-5.3.1 urdfdom_headers acceleration_firmware_kv260 ament_cuda isaac_ros_image_proc"

# fetch some additional missing dependencies and packages
# (...)

# build things
colcon build --merge-install 

Then custom built a launch script and launched it:

rectify_resize.launch.py

```bash import os from launch import LaunchDescription from launch_ros.actions import Node from launch_ros.substitutions import FindPackageShare from launch_ros.actions import ComposableNodeContainer from launch_ros.descriptions import ComposableNode from tracetools_launch.action import Trace from tracetools_trace.tools.names import DEFAULT_EVENTS_ROS from tracetools_trace.tools.names import DEFAULT_EVENTS_KERNEL from tracetools_trace.tools.names import DEFAULT_CONTEXT def generate_launch_description(): # Trace perception_container = ComposableNodeContainer( name="perception_container", namespace="", package="rclcpp_components", executable="component_container", composable_node_descriptions=[ ComposableNode( package="image_proc", plugin="image_proc::RectifyNode", name="rectify_node", remappings=[ ("image", "/camera/image_raw"), ("camera_info", "/camera/camera_info"), ], ), ComposableNode( namespace="resize", package="image_proc", plugin="image_proc::ResizeNode", name="resize_node", remappings=[ ("camera_info", "/camera/camera_info"), ("image", "/image_rect"), ("resize", "resize"), ], parameters=[ { "scale_height": 2.0, "scale_width": 2.0, } ], ), ], output="screen", ) return LaunchDescription([ # image pipeline perception_container ]) ```

Looks just equally good to me.

Moreover, further inspected QoS:

QoS of amd64 workstation topics:

ros2 topic info /camera/camera_info --verbose
Type: sensor_msgs/msg/CameraInfo

Publisher count: 1

Node name: camera_controller
Node namespace: /
Topic type: sensor_msgs/msg/CameraInfo
Endpoint type: PUBLISHER
GID: 01.0f.40.ad.32.2e.d7.5e.01.00.00.00.00.00.3f.03.00.00.00.00.00.00.00.00
QoS profile:
  Reliability: RELIABLE
  History (Depth): UNKNOWN
  Durability: VOLATILE
  Lifespan: Infinite
  Deadline: Infinite
  Liveliness: AUTOMATIC
  Liveliness lease duration: Infinite

QoS of accelerator (KV260)

ros2 topic info /resize/resize --verbose
Type: sensor_msgs/msg/Image

Publisher count: 1

Node name: resize_node
Node namespace: /resize
Topic type: sensor_msgs/msg/Image
Endpoint type: PUBLISHER
GID: 01.0f.cc.5b.10.94.f8.1f.01.00.00.00.00.00.2c.03.00.00.00.00.00.00.00.00
QoS profile:
  Reliability: RELIABLE
  History (Depth): UNKNOWN
  Durability: VOLATILE
  Lifespan: Infinite
  Deadline: Infinite
  Liveliness: AUTOMATIC
  Liveliness lease duration: Infinite

Subscription count: 1

Node name: rqt_gui_cpp_node_77964
Node namespace: /
Topic type: sensor_msgs/msg/Image
Endpoint type: SUBSCRIPTION
GID: 01.0f.40.ad.8c.30.94.40.02.00.00.00.00.00.17.04.00.00.00.00.00.00.00.00
QoS profile:
  Reliability: BEST_EFFORT
  History (Depth): UNKNOWN
  Durability: VOLATILE
  Lifespan: Infinite
  Deadline: Infinite
  Liveliness: AUTOMATIC
  Liveliness lease duration: Infinite

[vmayoral]

Overall @jasvinderkhurana, things look good to me while investigating the interoperability. I could not reproduce your situation above.

The only reasons that I can come up with for things to fail are:

1. BEST_EFFORT reliability in rqt is somehow clashing with RELIABLE reliability in the Nodes, this happened in past ROS distros/versions and led to interoperability issues but I believe it was addressed in Humble. I just confirmed it above.
2. You seem to be running things with an acceleration kernel, so maybe the kernel is not working? Have you tried subscribing to the topics and checking if images are being received?

Taking a look at your setup together may lead to a faster conclusion.

[jasvinderkhurana]

@vmayoral, thanks for looking into this, Just want to understand is your last experiment on KR260 board with 2022.1 tools?

And yes, looking together on my setup will be great, I can make the setup tomorrow and let's sync up.

[vmayoral]

@vmayoral, thanks for looking into this, Just want to understand is your last experiment on KR260 board with 2022.1 tools?

No, see title above of that section: "✅ Robotics perception: using KV260 accelerator with Ubuntu 22.04 image". I did not use Vitis to validate interoperability and I used the KV260 as my KR260 board is not usable right now.

[jasvinderkhurana]

Hi @vmayoral,

I tried again with KV260 board and now this time with the docker image that we discussed during our call. But still I am not able to see the intermediate images there. Please have a look below. Now really not sure what could be the difference.

[vmayoral]

That's terribly weird @jasvinderkhurana. Graph data is landing appropriately and you're clearly having interoperability "board-dev.station-board" because I can see the graph depicted there with Nodes from both entities.

My only guess regarding you not able to subscribe to topics from rqt is QoS-related. Feel free to schedule with me a call and let's look at your setup together.

Edit: Docker container is known to work and reproducible as demonstrated so my only guess is that there's something wrong on your setup that needs fixing.

[jasvinderkhurana]

@vmayoral, I am on business travel this week and won't have the hardware access. I will setup a call with you early next week.

FYI: Everyone who is trying this app is facing same issue.

[vmayoral]

Thanks for reporting @jasvinderkhurana. Unfortunately, as indicated above we have not able to reproduce your issues on our side on new machines and setups. We went further and Dockerized the robotics application to speed things up for reproduction. Then various of us (different persons, on different machines) have confirmed things internally, things just work out of the box.

On top of this, all data you've shared hints that there's nothing wrong on the KV260/KR260 so the only option left is that the issue is on your dev. machines (on the ROS 2 dev. machine side with Humble release). As adviced, a hands-on support session might be the way forward for those of you struggling to get basic things to work.

FYI: Everyone who is trying this app is facing same issue.

I'd be interested in learning more about this and who's facing this issue specifically. Happy to chat whenever you're back.

[vmayoral]

Progressed in here with Tomas, looked at his setup and realized that his ROS 2 and Gazebo installation isn't functional. He's unable to subscribe to basic topics from Gazebo (issue's not KRS related, but ROS 2 and Gazebo related).

We're still unable to reproduce the issues encountered. The way we're setting up new 22.04 machines with ROS 2 and Gazebo is as follows:

# set up sources and deps as described by official docs
# install ROS Desktop full with Ignition Gazebo
sudo apt-get install ros-humble-desktop-full
# install gazebo-ros-pkgs which pulls a compatible Gazebo Classic
sudo apt-get install ros-humble-gazebo-ros-pkgs

[jasvinderkhurana]

After reinstalling the packages it resolved the problem