Zenoh Bridge Limitations

[MarioAlmelaUMH]

Hello, I am currently working with Zatitech's Openplanner bridge (https://github.com/ZATiTech/open_planner). I've heard the Zenoh bridge is based on Zatitech's, and I was wondering if it has inherited the latter's limitations.

Currently, Openplanner bridge only supports one sensor each (one camera, one lidar, one GNSS and one IMU). Can you modify this sensor kit in Zenoh's bridge (for example, adding multiple cameras or lidars)?

Additionally, I've had problems when assigning each sensor a different _sensortick parameter (in order to change the frequency of each sensor), so right now the system only works with all sensors working at the same frequency. Is it possible to assign different sensor frequencies on Zenoh's bridge?

Thank you for your answers.

[evshary]

Hi

Hello, I am currently working with Zatitech's Openplanner bridge (https://github.com/ZATiTech/open_planner). I've heard the Zenoh bridge is based on Zatitech's, and I was wondering if it has inherited the latter's limitations.

in fact, Zenoh bridge is not based on Zatitech's. We rewrote the bridge, but using the map generated by Zatitech.

Currently, Openplanner bridge only supports one sensor each (one camera, one lidar, one GNSS and one IMU). Can you modify this sensor kit in Zenoh's bridge (for example, adding multiple cameras or lidars)?

It's possible to support multiple sensors. In Carla, your sensor has its own name, and the bridge will publish the ROS topic based on that name. For example, the bridge will publish camera's image with the topic "sensing/camera/{sensor_name}/image_raw". But I haven't tested multiple cameras on the same vehicle yet. If you have any problems, please let me know.

Additionally, I've had problems when assigning each sensor a different sensor_tick parameter (in order to change the frequency of each sensor), so right now the system only works with all sensors working at the same frequency. Is it possible to assign different sensor frequencies on Zenoh's bridge?

The frequency should be based on Carla setting, so theoretically each sensor should have its own frequency. However, I found that the frequency might not be as fast as you set. For example, if you set 30 FPS on camera, the bridge might only publish images at 15 Hz while there are several sensors. I'm not really sure if it's the performance limitation on Carla or if there is anything wrong with the bridge. Need more investigation.

[MarioAlmelaUMH]

Hello, does this bridge support different sync modes? Currently I'm working with Carla's Synchronous mode, is Asynchronous also supported?

Thank you.

[evshary]

Hello, does this bridge support different sync modes? Currently I'm working with Carla's Synchronous mode, is Asynchronous also supported?

Thank you.

No, the bridge only supports synchronous mode now.

[MarioAlmelaUMH]

I was wondering, is it possible to change the Carla map that the bridge uses? Currently it automatically sets Town01, is there a config option to change this, add more vehicles (Non-ego, Carla controlled), change the ego vehicle's initial position, etc.?

Also, I've tried to run ros2 topic list inside the docker container, but only topics /parameter_events and /rosout show up. Is there a way to see Autoware topics?

Thank you for your answer.

[evshary]

I was wondering, is it possible to change the Carla map that the bridge uses? Currently it automatically sets Town01, is there a config option to change this, add more vehicles (Non-ego, Carla controlled), change the ego vehicle's initial position, etc.?

To change the map, please refer to the link https://github.com/evshary/autoware_carla_launch/tree/humble/carla_map For Carla settings, you can revise the Python script here https://github.com/evshary/zenoh_carla_bridge/tree/4413e2907c807826c4e3f3df1e4aec74a5353774/carla_agent To change the initial pose, you can add the argument to the Python script https://github.com/evshary/autoware_carla_launch/blob/humble/script/run-bridge-two-vehicles.sh#L13

Also, I've tried to run ros2 topic list inside the docker container, but only topics /parameter_events and /rosout show up. Is there a way to see Autoware topics?

Remember to run source env.sh before ros2 command. However, since the bridge sends Zenoh data to zenoh-bridge-dds, you can't run ros2 topic list to get the messages from the bridge unless you've already run Autoware in your container. I would suggest using tmux or other terminal multiplexers in the container. Then you can run Autoware in one terminal and run the ros2 command in another one to see the detail.

[MarioAlmelaUMH]

Thank you so much. I've followed this steps and now I'm able to see the complete list of topics. I still have a couple issues, however.

• When I try to echo a topic, it throws an error as the Autoware specific message type cannot be determined. I have tried to source the environment with source env.sh, but it doesn't seem to work when using tmux (this can be seen in the right panel in the image below). I already sourced the environment on the left panel before starting the tmux session:

I need to echo the topics so I can check if the modules are working properly.

• On another topic, I may want to change the ego vehicle's sensor kit, to include multiple cameras and lidars. I suppose there will be 2 config files, one for Carla and one for Autoware, and both need to have the same sensor config. Is this right? Which files should I modify?

Thank you so much for your answers.

[evshary]

For the first one, this is because in tmux $0 is -bash not bash. For the temporary workaround, please run bash after splitting a new terminal in tmux. I'll see how to fix it.

About the second topic, you're right. You need to add extra sensors for the vehicle in Carla by yourselves https://github.com/evshary/zenoh_carla_bridge/blob/main/carla_agent/simulation/world.py#L118 For the Autoware, you can refer to the official documentation https://autowarefoundation.github.io/autoware-documentation/main/how-to-guides/integrating-autoware/creating-vehicle-and-sensor-description/creating-vehicle-and-sensor-description/ The sensor_kit_description and sensor_kit_launch are here https://github.com/evshary/autoware_carla_launch/tree/humble/src

[MarioAlmelaUMH]

Thank you. I've been able to echo some topics and source the environment properly. I'm trying to record a ros2bag using this command: ros2 bag record /sensing/camera/traffic_light/image_raw /sensing/imu/tamagawa/imu_raw /sensing/lidar/concatenated/pointcloud /sensing/gnss/pose_with_covariance /perception/object_recognition/detection/clustering/clusters /perception/object_recognition/detection/clustering/objects /perception/object_recognition/detection/objects /perception/object_recognition/objects /system/component_state_monitor/component/autonomous/perception /control/command/control_cmd /perception/object_recognition/detection/rois0 perception/object_recognition/detection/camera_lidar_fusion/objects /clock -o <bag_name> --qos-profile-overrides-path qos_override.yaml

However, when I do this, the following error appears:

The ros2bag isn't even subscribing to this topic, yet it seems it has two types associated to it. I have never encountered this error before, so I have no clue as to what may be happening.

Thank you for your answers.

[evshary]

Thanks for your report. I can reproduce the issue but might need further investigation.

[MarioAlmelaUMH]

Hello,

• First of all, is there any new insight regarding the ros2 bag issue?
• Secondly, I have tried to activate camera_lidar_fusion mode for perception module, however the camera related topics are not receiving messages (perception/object_recognition/detection/rois* are all empty). The console log shows these: Apparently, camera info topics are not being published. I've de-commented the camera driver in sensing.launch.xml file, but still nothing. I've looked at the tier4_perception_component.launch.xml and other related files, but I cannot find the cause of this issue.

Thank you so much for your answers.

[evshary]

I think the first issue comes from Autoware itself. Since I fixed at old version and that version has some topic type conflicts on /perception/traffic_light_recognitio n/traffic_signals. You can check with ros2 topic info -v /perception/traffic_light_recognition/traffic_signals. I think the issue will be fixed in this commit, so the Autoware we use should be upgraded. I'm not sure whether there is other issue for newer Autoware, so not my first priority to upgrade now.

[evshary]

For the second one, I publish the camera_info with this name /sensing/camera/traffic_light/camera_info by default However, if there is no any subscriber, then you can't see it. Maybe you can check which topic name camera_lidar_fusion is subscribing.

[MarioAlmelaUMH]

Thank you for your answers. Regarding the second question, I've checked the file camera.launch.xml and I've seen the output topic of the camera as sensing/camera/traffic_light/camera_info.

The camera_lidar_fusion node listens to topics of the form sensing/camera/camera*/camera_info, with (*) being the number of the camera. I changed the topic in camera.launch.xml manually to sensing/camera/camera0/camera_info, but it doesn't seem to work. There are deeper Autoware files like perception.launch.xml, which define in more depth this camera topics. However, I can't find them in my current installment of the bridge. I've tried searching outside and inside the docker container, but I still haven't found these files. Where is Autoware proper located inside the bridge?

Thank you.

[evshary]

You also need to change the camera name from Python script https://github.com/evshary/zenoh_carla_bridge/blob/main/carla_agent/simulation/world.py#L124 Then the bridge will use your name to publish the topic.

For the Autoware, it's a little tricky. I found that the current Autoware container doesn't include the source code (Only binary). It's not convenient to do development, but you can try to clone your own Autoware and put it into the container. No need to clone the whole Autoware. You can only get part of the component you want to modify in Autoware and overlay the original one.

[MarioAlmelaUMH]

Thanks for your answer. I'm looking at the makefile and the dockerfiles of the bridge, and I don't know where exactly should I put the clone statement. In those files, I haven't seen the instruction that introduces the binary Autoware files inside the container (unless they are already inside the image that gets downloaded when you first run the bridge). I'm not very experienced yet with docker and docker images. How could I clone the original Autoware inside the container? I'm guessing I should add the clone command to one of the dockerfiles then rebuild the bridge, but how can I override the current binary files?

Thank you .

[evshary]

The Autoware binary files are already in the base image (ghcr.io/autowarefoundation/autoware-universe:humble-20230715-prebuilt-cuda-amd64). You can look at /autoware folder, which includes the binary files.

To overlay the existing Autoware packages, you can find out which package you want to use first and put it under autoware_carla_launch/src folder. After rebuilding the project, you'll use the package cloned by yourself.

You can refer to here. I overlay the autoware_launch by myself to avoid some issues.

[datvuthanh]

Hi @MarioAlmelaUMH , i ran this script source env.sh but it didn't work. Could you give me some suggestions?

[MarioAlmelaUMH]

I'm not an expert on the matter either. For me, it has always worked fine. What kind of problem did you have?

[datvuthanh]

Haha sorry, I commented wrong issue :D, my issue here: https://github.com/evshary/autoware_carla_launch/issues/25

[evshary]

@MarioAlmelaUMH I added some script for building your own Autoware, but now only supports Carla 0.9.14 (Need to port back to Carla 0.9.13) https://github.com/evshary/autoware_carla_launch/tree/humble-0.9.14#build If you're still struggling with building Autoware in the docker, maybe you can try this one.

By the way, I found that we would face some CUDA issues while building with the prebuilt image (humble-2023.08-prebuilt-cuda-amd64), so I used another image (humble-2023.08-cuda-amd64) instead. Also, I use 2023.08 instead of 2023.07 to make the code consistent. The date of the code in tag 2023.07 is 2023.07.29, but the code in prebuilt binary is 2023.07.15. It'll cause some conflicts.

[MarioAlmelaUMH]

Ok, thank you. I still have to try the first option, but if that gives me trouble, I will try out this new one. I will also take into account the images used. I have to run cuda so better to start with humble-2023.08-cuda-amd64. I will keep updating as I discover new things.

[MarioAlmelaUMH]

Hello, I've been trying to incorporate Autoware's source code into the container, and I've run into some issues. I know that the newest version of the bridge may solve all of this, but I still want to explain my process in case I did something wrong. The version of the bridge I'm using comes from this commit: 5d51a5b5e0f0b9ee5ac3572c78e3a4a8b1bdd5c2

• First, I downloaded Autoware-2023.08 from the official repository (https://github.com/autowarefoundation/autoware/tree/2023.08). I moved the resulting folder inside the bridge's root directory (inside de autoware_carla_launch folder)
• Then, I modified the Dockerfile to download the image humble-2023.08-cuda-amd64, instead of the 20230715 one
• I run the run-autoware-docker.sh file, and it downloaded the proper image and built the container without issue
• Once inside the container, I followed Autoware's installation guide to build Autoware inside of the container. I also made sure to add the autoware_carla_launch node (autoware_carla_launch/src/autoware_carla_launch) to Autoware's src folder (autoware_carla_launch/Autoware-2023.08/src/autoware_carla_launch). This way it wouldn't throw an error when I run run-autoware.sh file
• The building process was successful. After that, I sourced the new Autoware's environment instead of the bridge's one (source Autoware-2023.08/install/setup.bash)
• When I run run-autoware.sh file, I got the following error:

This error is not new. It happened before when I tried to add Autoware's source code inside of the image. Basically it's asking for a series of parameters (starting with pose_source) that Autoware needs but are not specified by the bridge. I solved a couple of these issues by hard-coding a default value for these parameters inside Autoware's launch files, but the list of parameters that needed this was too long and I couldn't find the proper default values on the internet.

My hypothesis is that some file is missing that maps or gives value to all of this parameters. The bridge doesn't need this file because usually the Autoware docker image already has Autoware prebuilt, and now it's missing because I have to use the bridge's launch node instead of the one from Autoware. This could be wrong, it is just my head cannon on what's happening.

Is there anything wrong with the process I followed? Could it be viable to build Autoware's source code without having to update the bridge version? There's multiple people working with the bridge and we all should work with the same version, so for the sake of consistency, I would appreciate not to update the bridge version unless is completely necessary.

Thank you so much for your answers and best regards, Mario

[evshary]

Hi @MarioAlmelaUMH, I believe the key problem here is that you also need to bump the version of autoware_launch to version carla_202308. Here is the commit I did: https://github.com/evshary/autoware_carla_launch/commit/467b7c91b5d1d1ff5fc8297322dda80d417ef228#diff-66a6d5b656005e10aade7bf7a5c9c7722252f3800d4be3f0e5cf14f4847432cc

I picked the commits for Carla 0.9.13, and pushed them to the branch support_src_for_humble I haven't tested it properly yet (traveling now), but maybe you can try that and see if it works for you or not. https://github.com/evshary/autoware_carla_launch/tree/support_src_for_humble#build-the-container-for-zenohautoware If everything is ok, I'll merge it into the humble branch

[MarioAlmelaUMH]

Hi @evshary . I have tested both options you showed me, but unfortunately they haven't worked yet. I'll start with the last option. I visited the github page you showed (https://github.com/evshary/autoware_carla_launch/tree/support_src_for_humble#build-the-container-for-zenohautoware) and followed the instructions of the optional source installation part. I run the script build-autoware.sh and the proceeded as told with make prepare_autoware and make_build_autoware. Everything seemed fine, but when I tries to run the bridge as normally, this error popped up:

I'll include the log files for more detail: zenoh_bridge_dds.log autoware.log

Basically, I think the carla_autoware_launch node is not being found. I have tried to rebuild the bridge a couple times, but I'm not sure if it is building correctly. Also, aparently there's a config file whose path is missing.

---

On another note, for the first option, I bumped the autoware_launch version to carla_202308, but this didn't work either. I had to copy and paste multiple folders into Autoware (carla_pointcloud, carla_sensor_kit_description and carla_sensor_kit_launch), as well as modify the env.sh file to source Autoware's setup.bash instead of the bridge's.

With all, I was able to launch Autoware, but the ROS Time was frozen at 0 and the Autoware dds didn't seem to be connected to the zenoh_carla_bridge. After I tried the second option (https://github.com/evshary/autoware_carla_launch/tree/support_src_for_humble#build-the-container-for-zenohautoware) I was not able to rebuild the first container again and reproduce the issue, being stuck in this new error: I have deleted and rebuilt Autoware-2023.08, deleted and rebuilt the docker image and cleaned the Autoware container with make clean_autoware, but still, I have not yet been able to get Autoware to open up again.

[evshary]

@MarioAlmelaUMH For the first question, can you check whether you run source env.sh or not? I believe you are missing some variables. https://github.com/evshary/autoware_carla_launch/blob/support_src_for_humble/env.sh#L13 Remember you still need to run source env.sh after building.

For the second question, I believe the package autoware_carla_launch is not built. Can you check whether it exists in install folder?

[MarioAlmelaUMH]

Thanks for your answer, @evshary . I tried again the first option (following this link: https://github.com/evshary/autoware_carla_launch/tree/support_src_for_humble#build-the-container-for-zenohautoware) and this time it worked. We are now exploring Autoware's source code in order to find how can we modify the sensor kit.

Since that last part belongs to this thread (https://github.com/evshary/autoware_carla_launch/issues/43), I think it's safe to finally close this one. I'll reopen it if more questions come afloat, but for now, I think it's ok.

Thank you so much.