Sensor kit modification not showing up in topic list

[MarioAlmelaUMH]

Hello, I already mentioned this issue in https://github.com/evshary/autoware_carla_launch/issues/32, but I think it will be more clear if I open a thread specifically for it.

I'm trying to modify the default sensor kit included with the bridge. Apparently, Autoware should support up to 8 cameras and 8 lidars. I have modified the file world.py and added more cameras and lidars. These extra sensors are then detected by run-bridge.sh. However, when I run run-autoware.shI can't see the new topics corresponding to those extra sensors.

I have checked sensor_kit_calibration.yamland sensor_kit.xacro to include them with the same names as world.py, but still can't see the topics when I run ros2 topic echo. Everything looks as default.

I think I'm missing a step here. Should I modify other files? Am I missing something?

Thank you for your answers.

[evshary]

Do you add your own camera in the Python script? https://github.com/evshary/zenoh_carla_bridge/blob/main/carla_agent/simulation/world.py#L124 The bridge will only bridge the sensors existing in Carla, which is created by Python script.

[MarioAlmelaUMH]

Yes, I tried to modify the script world.py by adding more cameras. One thing I also don't understand is the CameraManager class. Do I have to add one per camera, or just using a single one for all cameras is OK? I'm planning to assign each camera a different position though.

Thank you for your answers.

[evshary]

Yes, I tried to modify the script world.py by adding more cameras. One thing I also don't understand is the CameraManager class. Do I have to add one per camera, or just using a single one for all cameras is OK? I'm planning to assign each camera a different position though.

Thank you for your answers.

No, I believe you should add rgb_camera member under World class. You can use the same RgbCamera class, but need to pass the position argument into the class object. (Need some modification)

[MarioAlmelaUMH]

Yeah, right now I was duplicating the line where the RgbCamera was instantiated and changing the class variable so it doesn't overwrite, like:

self.rgb_camera = RgbCamera(self.player, sensor_name='traffic_manager') self.rgb_camera2 = RgbCamera(self.player, sensor_name='camera0')

With this setup, the run-bridge.sh detected the two cameras, but that's it. I couldn't see any image_raw topics besides the one by default.

[MarioAlmelaUMH]

Hello, we’ve been trying to add new sensors to the default sensor kit of the Zenoh bridge, but the new sensors we added are not detected by Autoware. They had no matching Autoware topic that we could monitor.

In order to track this problem, we added one additional camera to the bridge (using the world.py file), called “camera_test”, and then defined a series of points of interest alongside the bridge, as shown in the image below. The objective behind this was to track where the signal from camera_test is lost.

Point 0 corresponds to Autoware, where we can’t find the topics related to camera_test. Points 1 and 2 correspond to Carla itself, as well as the callbacks found in the sensor_bridge.rs file. We could successfully identify the two topics associated with cameras traffic_light and camera_test in register_camera_rgb function of the sensor_bridge.rs, and both were successfully published with data.

In Point 3, we tracked the connection between the zenoh_bridge_dds and Autoware itself, and found the Route Zenoh→DDS and Route DDS→Zenoh classes. By examining the zenoh_bridge_dds log, we found the matches between Zenoh topics and DDS topics. In these logs, we observed that the doute Zenoh-DDS was successfully discovered and created for the traffic_light camera (the one by default), but none for the camera_test camera (the one that we added).

At this point, we understand that new camera data worked fine in the Carla side of the bridge (Carla itself and its callbacks) but was somehow lost before the Zenoh topics got matched to DDS topics. This led us to investigate Point 4, i.e. the connection between both parts of the bridge and the publishing and subscribing processes of the Zenoh topics.

At this point, we have two hypothesis: first, we thought that maybe both cameras were publishing to the same topic. Despite being two topics corresponding to the two cameras as per Point 2, those two topics were being mapped to the same DDS topic. To test this hypothesis, we prepared a scenario with two cameras (traffic_light and camera_test) with a frequency of 50 ms each and moves the vehicle around Autoware while recording Autoware’s sensing/camera/traffic_light/image_raw topic through ros2bag. If the 2 cameras were publishing on the same topic, we should see pairs of messages with identical timestamp, as both cameras share the same frequency. However, the recordings showed a single message per timestamp tick, which proved that one of the cameras was being ignored.

The second hypothesis was that the Zenoh topic corresponding to the new camera (/v1/rt/sensing/camera/camera_test/image_raw) wasn’t being subscribed by the zenoh_bridge_dds. We investigated Zenoh’s discovery process and Zenoh to DDS routing. We tried to understand how the zenoh_bridge_dds detects sensor topics, and why it can detect the traffic_light camera topic but can’t detect the camera_test topic. We reviewed route_zenoh_dds.rs, lib.rs and ros_discovery.rs files, among others. The run_local_discovery method is not able to discover the topic related to the new camera. The try_add_route_from_dds function was not called to add the topic related to the new camera. The new route is not created in RouteZenohDDS.

In summary, at this point, we see that the zenoh_carla_bridge successfully discovers the new camera topics and apparently is able to publish messages towards the zeno_bridge_dds. However, the zeno_bridge_dds is not able to discover this new topic.

We would appreciate any feedback to try to solve this issue.

Thank you

Mario

[evshary]

Hi Mario, great analysis!

Sorry, maybe I didn't make you clear in the last call. Yes, zenoh_bridge_dds can't detect the topic because there is no corresponding subscriber on the Autoware side (that is 0 in your case). Although you might have tried ros2 topic echo /sensing/camera/camera_test/image_raw before, ROS doesn't provide enough information (like ROS type) for zenoh-bridge-dds to create the corresponding publisher to publish ROS 2 topic. To solve the issue, there are two ways:

1. Create a dummy subscriber on the Autoware side. You can write your own ROS node to subscribe topic or maybe use some existing packages

# In the Autoware container
sudo apt update
sudo apt install ros-humble-image-tools
ros2 run image_tools showimage --ros-args -r image:=/sensing/camera/camera_test/image_raw

2. Find out why Autoware doesn't subscribe to the camera topic. I haven't taken a deeper look at the sensor fusion of camera and lidar data in Autoware, but if Autoware can subscribe /sensing/camera/camera_test/image_raw, then you should also be able to see the topic.

Let me know if there is anything unclear to you. Thanks!

[MarioAlmelaUMH]

Dear ChenYing, Thank you very much for your answer.

I have tried the image_tools library you proposed and I successfully visualized the images published on the /sensing/camera/camera_test/image_raw. However, I was not able to record the images using ros2bag. Additionally, I'd need to know whether these images are being processed by Autoware, or we are just monitoring a topic externally. I’m sorry if this point wasn’t properly clarified, but it’s not enough for us to see the images being generated by the cameras, because we also want those images to enter Autoware and be processed like the images from the traffic_light camera. Are the images shown with image_tools being processed by Autoware?

Secondly, I could understand if there were problems with /sensing/camera/camera_test/image_raw topic, as it’s not a native Autoware topic (the camera_test name is not present in Autoware’s code). However, we also tested names like camera0 (a camera name that matches exactly to one of Autoware’s topics) and the /sensing/camera/camera0/image_raw topic was still missing from ros2 topic list. I want to clarify that the only topic to appear in ros topic list is /sensing/camera/traffic_light/image_raw, the other sensor’s image_raw topics don’t show up on ros topic list, independently of the sensor’s name.

I don’t know if these points will help, but I think they were necessary in order to clarify things.

We will now explore Autoware itself and see if we can make it work by linking the source code to the Zenoh bridge (instead of linking a Docker container with its binary).

Thank you so much for your answers.

Best regards, Mario.

[evshary]

Hi @MarioAlmelaUMH

For the first question, no. The topic is not processed by Autoware now. Now it's just published from Carla and the bridge forward to the other side. If you want to record the camera data with rosbag, you still need a subscriber to subscribe camera topic to make the bridge forward camera data. However, I think that's not what you need because Autoware doesn't do anything to it.

For the topic /sensing/camera/camera0/image_raw, I believe that Autoware doesn't subscribe to this topic by default. I've also checked the AWSIM case, and it doesn't subscribe to the topic either. I've taken a quick look at how Autoware uses camera0 data. I think it's related to Yolo objection detection, which is related to the discussion you raised on Autoware GitHub. https://github.com/orgs/autowarefoundation/discussions/3687#discussioncomment-6544216

For running with Autoware source code, feel free to use the following link I mentioned before. https://github.com/evshary/autoware_carla_launch/tree/humble-0.9.14#build But remember that this only works on Carla 0.9.14.

[MarioAlmelaUMH]

Hi @evshary

I wanted to ask about an odd situation I encountered while working with the bridge that includes Autoware's source code.

I recall that, when I used the normal prebuilt version, I changed the perception_mode in autoware.launch.xml file (the one inside autoware_carla_launch/src/autoware_launch) to camera_lidar_fusion and that was enough for the camera_info topics to appear in ros2 topic list (/sensing/camera/camera*/camera_info, with * = [0-5]). They were not receiving any data, but they appeared nonetheless.

However, after installing the source compatible version (https://github.com/evshary/autoware_carla_launch/tree/support_src_for_humble), I also changed the perception_mode parameter the same way but the camera_infotopics didn't show up.

I was curious, so I downloaded the source code for the 2023.07 version of Autoware (the version used by the prebuilt image I had) and compared it to the source code of the support_src_for_humble bridge (which is version 2023.08). I compared both versions, but couldn't find any clues as to what happened. Most of the camera related files where the same and the ones that changed didn't have camera related changes. That makes it really weird that in the prebuilt version the camera_info topics showed up but in the source version didn't, as both versions of Autoware have basically the same camera code.

• Do you know if the prebuilt image version did something else to Autoware that could explain this (some extra processing steps, etc.)?

• Do you think it's maybe related to the sensor_kit package?

Thank you for your answers. Best regards, Mario

[evshary]

Hi Mario, Maybe you can test with the prebuilt image 2023.08, which is also available in humble now. I don't know too much about whether the prebuilt image does anything special, but if the prebuilt image 2023.08 also has the same problem, then it might be related to the changes between 2023.07 and 2023.08.

[MarioAlmelaUMH]

Hi, @evshary . We finally solved it. We are now able to add multiple cameras, they appear on the ros2 topic list and also receive proper data. As you explained before, there was a missing subscriber, so the camera topics were not created. Now we have added the subscriber and we can see the different cameras properly.

Now, our focus is to do the same for radars and lidars, and ensure all data is properly fed to the perception module.

For the time being, I'm going to mark this issue as completed. I'll reopen it if more problems arise, but for now it's complete.