Issues with multiple vehicles

[zeidk]

Hello, I followed the instructions to start 2 vehicles in 2 different rviz windows. This part is working.

When I do ros2 topic list, I only get: /parameter_events and /rosout. Is this expected? Moreover, in RViz, when I set a 2D pose goal, the path is computed but engage does not move the vehicle, it is still in the INITIALIZING state.

Any help will be useful.

Thank you!

Note: I am not using docker and I am using the galactic branch.

[Kim-mins]

For ros2 topic, this link would help you: https://github.com/tier4/AWSIM/issues/81#issuecomment-1425102055 And for 2d pose goal, maybe the 'INITIALIZING' is the status of Localization of RViz, is it right? If so, you can set a goal after the status being switched to 'INITIALIZED'(with green background).

[evshary]

Are you following the tutorial here? https://github.com/evshary/autoware_carla_launch/tree/galactic#run-multiple-vehicles-with-autoware-in-carla-at-the-same-time Could you check whether ROS_DOMAIN_ID=1 ros2 topic list and ROS_DOMAIN_ID=2 ros2 topic list works well or not? Since we are using zenoh-bridge-dds here, you can not run ros2 topic list directly in Zenoh network, but it should work in Autoware DDS domain.

For the INITIALIZING part, maybe you can capture the printscreen or record the video, it'll help us to check the problem. I guess the localization failed in your scenario.

By the way, is the single vehicle scenario successful in your environment? Just want to check whether this is an issue from multiple vehicles.

[zeidk]

Video link: https://drive.google.com/file/d/1WtYmNneMDSoTDUn8MvDF7aGJIQSmB4wh/view?usp=sharing

@Kim-mins Thank you for your reply. I am not using docker and it seems that the link you provided is for docker. Should I still try these commands even outside docker? In the attached the video, the vehicle is not initialized. A plan is generated when I choose a goal but when I press Engage, nothing happens.

@evshary Yes, I followed the instructions for galactic and modified the correct files. Thank you, the command ROS_DOMAIN_ID=1 ros2 topic list gives me the list of topics (as seen in the video). How to I specify the ROS_DOMAIN_ID when I want to subscribe and publish to topics using ROS 2 python? To answer your question, the scenario with a single vehicle works (ros2 launch autoware_carla_launch autoware_carla.launch.xml)

Below is the log from launching the bridge. Do you think WARNING: Version mismatch detected: You are trying to connect to a simulator that might be incompatible with this API may be the issue? `

INFO] [launch]: All log files can be found below /home/zeid/.ros/log/2023-06-26-19-33-10-488464-zeid-MAX-15-74282 [INFO] [launch]: Default logging verbosity is set to INFO [INFO] [zenoh_carla_bridge-1]: process started with pid [74284] [INFO] [poetry-2]: process started with pid [74286] [INFO] [poetry-3]: process started with pid [74288] [zenoh_carla_bridge-1] WARNING: Version mismatch detected: You are trying to connect to a simulator that might be incompatible with this API [zenoh_carla_bridge-1] WARNING: Client API version = 0.4.0-1-gb3fbf09-dirty [zenoh_carla_bridge-1] WARNING: Simulator API version = 4049780d [zenoh_carla_bridge-1] WARNING: Version mismatch detected: You are trying to connect to a simulator that might be incompatible with this API [zenoh_carla_bridge-1] WARNING: Client API version = 0.4.0-1-gb3fbf09-dirty [zenoh_carla_bridge-1] WARNING: Simulator API version = 4049780d [poetry-2] INFO: listening to server 127.0.0.1:2000 [poetry-3] INFO: listening to server 127.0.0.1:2000 `

[evshary]

@zeidk You can run export ROS_DOMAIN_ID=1 before running ROS 2 Python. For the warning message WARNING: Version mismatch detected: You are trying to connect to a simulator that might be incompatible with this API, I don't think it's the root cause, because I can run successfully with this message.

I've tried several times to reproduce your problem, but still unable to reproduce. I've updated the galactic branch with some fixes. Could you fetch the code, rebuild and try again?

[zeidk]

@evshary Thank you for the help. Unfortunately I still have the same issue. Do you mind sending me your autoware_zenoh.launch.xml? I feel like you have different values which I need to replicate in my file.

Also, autoware.launch.xml has the following argument <arg name="vehicle_id" default="$(env VEHICLE_ID default)" description="vehicle specific ID"/>. Do I need to provide this argument in autoware_zenoh.launch.xml when working with multiple vehicles?

Thank you

[evshary]

I don't think autoware_zenoh.launch.xml has any difference. I used the default one.

<?xml version="1.0" encoding="UTF-8"?>
<launch>
  <!-- Essential parameters -->
  <arg name="map_path" default="$(env CARLA_MAP_PATH)" description="point cloud and lanelet2 map directory path"/>
  <arg name="vehicle_model" default="sample_vehicle" description="vehicle model name"/>
  <arg name="sensor_model" default="carla_sensor_kit" description="sensor model name"/>

  <!-- Optional parameters -->
  <!-- Modules to be launched -->
  <arg name="carla_interface" default="true" description="convert carla sensor data to autoware suitable format"/>
  <arg name="vehicle" default="true" description="launch vehicle"/>
  <arg name="system" default="true" description="launch system"/>
  <arg name="map" default="true" description="launch map"/>
  <arg name="sensing" default="true" description="launch sensing"/>
  <arg name="localization" default="true" description="launch localization"/>
  <arg name="perception" default="true" description="launch perception"/>
  <arg name="planning" default="true" description="launch planning"/>
  <arg name="control" default="true" description="launch control"/>
  <!-- Map -->
  <arg name="lanelet2_map_file" default="lanelet2_map.osm" description="lanelet2 map file name"/>
  <arg name="pointcloud_map_file" default="pointcloud_map.pcd" description="pointcloud map file name"/>
  <!-- Control -->
  <arg name="enable_obstacle_collision_checker" default="false" description="use obstacle_collision_checker"/>
  <!-- Vehicle -->
  <arg name="launch_vehicle_interface" default="false"/>
  <!-- System -->
  <arg name="launch_system_monitor" default="false" description="launch system monitor"/>
  <!-- Tools -->
  <arg name="rviz" default="true" description="launch rviz"/>
  <arg name="rviz_config" default="$(find-pkg-share autoware_launch)/rviz/autoware.rviz" description="rviz config"/>

  <group scoped="false">
    <include file="$(find-pkg-share autoware_launch)/launch/autoware.launch.xml">
      <!-- Common -->
      <arg name="map_path" value="$(var map_path)"/>
      <arg name="vehicle_model" value="$(var vehicle_model)"/>
      <arg name="sensor_model" value="$(var sensor_model)"/>
      <!-- Modules to be launched -->
      <arg name="launch_vehicle" value="$(var vehicle)"/>
      <arg name="launch_map" value="$(var map)"/>
      <arg name="launch_sensing" value="$(var sensing)"/>
      <arg name="launch_localization" value="$(var localization)"/>
      <arg name="launch_perception" value="$(var perception)"/>
      <arg name="launch_planning" value="$(var planning)"/>
      <arg name="launch_control" value="$(var control)"/>
      <!-- Global parameters -->
      <arg name="use_sim_time" value="true"/>
      <!-- Vehicle -->
      <arg name="launch_vehicle_interface" value="$(var launch_vehicle_interface)"/>
      <!-- Map -->
      <arg name="lanelet2_map_file" value="$(var lanelet2_map_file)"/>
      <arg name="pointcloud_map_file" value="$(var pointcloud_map_file)"/>
      <!-- Control -->
      <arg name="enable_obstacle_collision_checker" value="$(var enable_obstacle_collision_checker)"/>
      <!-- System -->
      <arg name="launch_system_monitor" value="$(var launch_system_monitor)"/>
      <!-- Sensing -->
      <arg name="launch_sensing_driver" value="false"/>
      <!-- Perception-->
      <arg name="traffic_light_recognition/enable_fine_detection" value="false"/>
      <!-- Tools -->
      <arg name="rviz" value="$(var rviz)"/>
      <arg name="rviz_config" value="$(var rviz_config)"/>
    </include>
  </group>

  <!-- CARLA -->
  <group if="$(var carla_interface)">
        <node pkg="carla_pointcloud" exec="carla_pointcloud_node" name="carla_pointcloud_interface" output="screen"/>
    </group>

  <!-- Bridge -->
  <group>
    <executable cmd="$(env AUTOWARE_CARLA_ROOT)/external/zenoh-plugin-dds/target/release/zenoh-bridge-dds -s $(env VEHICLE_NAME) -d $(env ROS_DOMAIN_ID) -c $(env ZENOH_BRIDGE_DDS_CONFIG)" output="screen" />
  </group>
</launch>

Also, autoware.launch.xml has the following argument . Do I need to provide this argument in autoware_zenoh.launch.xml when working with multiple vehicles?

No, I don't think it's required. I've searched the autoware folder and only see vehicle_interface.launch.xml uses it, but it seems like the launch only contains an argument.

<?xml version="1.0" encoding="UTF-8"?>
<launch>
  <arg name="vehicle_id" default="$(env VEHICLE_ID default)"/>
</launch>

By the way, are you using the latest galactic Autoware?

[zeidk]

Yes, I am using galactic. I reinstalled both autoware and your package, edited the files to be able to run both vehicles but I still have the same issue. I don't know where else to look...

I tried to look into the autoware code to see where the initialization happens but I found was not very helpful. Do you think one of the sensors may not work properly with two vehicles?

ROS_DOMAIN_ID=1 ros2 topic echo /initialpose shows me that the initial pose for v1 is published but it is still initializing in rviz.

Can I ask you how you installed Carla? Are you using the pre-built version?

Edit: It looks like --sync solves the issue. Is it because my computer is too slow? Are there any options I can pass to the following commands to speed up the process?

ROS_DOMAIN_ID=1 VEHICLE_NAME="v1" ros2 launch autoware_carla_launch autoware_zenoh.launch.xml ROS_DOMAIN_ID=2 VEHICLE_NAME="v2" ros2 launch autoware_carla_launch autoware_zenoh.launch.xml

[evshary]

I use the prebuilt Carla on GitHub https://github.com/carla-simulator/carla/releases/tag/0.9.13

About adding --sync, that's a little strange, because it just means ticking the Carla simulator periodically. And I've already done it in here.

Maybe you can modify 100 to 200 in thread::sleep(Duration::from_millis(100)); to slow down the simulator and see whether it works or not. (Remember to rebuild) BTW, which kind of GPU are you using?

[zeidk]

@evshary Thank you, I will try to change the duration and I will report back.

What is the difference between 0.9.13 and 0.9.13RSS?

The GPU I am using is GeForce RTX 2080 SUPER Mobile / Max-Q

I would like to run Carla on and zenoh/autoware on different machines. Do I only need to modify the port and the ip address to be able to do this? From my understanding, below is what I should do, did I miss anything?

• Carla: Use an open port
• Zenoh/Autoware: Use the same port as Carla and the ip address of the machine on which Carla is running.

[evshary]

RSS is related to safety, but I didn't use that. https://carla.readthedocs.io/en/latest/adv_rss/

For the GPU, I'm not sure whether this is the key point, I'm using GeForce RTX 3060. But I found that there is room for performance improvement in my bridge. I'll try to restructure that.

I would like to run Carla on and zenoh/autoware on different machines. Do I only need to modify the port and the ip address to be able to do this? From my understanding, below is what I should do, did I miss anything?

In fact, my bridge goes with Carla (connecting to 127.0.0.1:2000), so the architecture should be that

For the connection between Carla/bridge and Zenoh/Autoware, we use Zenoh. If your two machines are in the same LAN, they can connect to each other automatically.

[zeidk]

I am going to switch to humble. Should I use the humble branch or the main branch?

[evshary]

I am going to switch to humble. Should I use the humble branch or the main branch?

You can use the humble branch. The big difference in the humble branch is that we use containers to avoid environment setup issues.

[zeidk]

Hello, I am still on galactic. I have attached the log file associated to the run of the bridge (ros2 launch autoware_carla_launch carla_bridge.launch.xml)

During the bridge execution, some actors are created and then deleted. I'm not sure if this is expected. There are also messages of usupported types. Again, I don't know if this is expected or not.

Thank you.

bridge_log.txt

[zeidk]

I will be closing this ticket, I would like to provide some feedback to others who may have the same issue. After spending some time on this issue, there are two main reasons for not being able to run multiple vehicles:

• The first one is related on how carla is run. After multiple tries, the only way that worked for me is: /CarlaUE4.sh -quality-level=Epic -prefernvidia -world-port=2000 -RenderOffScreen. Multiple attempts failed because I did not provide -prefernvidia
• The second issue is due to not using a computer powerful enough to spawn multiple vehicles. In my case I start rviz for the first vehicle but not for the second vehicle. I initialize the vehicles and send goals to the vehicles by writing my own node and I do not use any of the RViz interface (e.g., 2D pose estimate).

Before I close this ticket, do you mind telling me if it's possible to spawn more than two vehicles if I want to? Is your package limited to only two vehicles?

[evshary]

During the bridge execution, some actors are created and then deleted. I'm not sure if this is expected. There are also messages of usupported types. Again, I don't know if this is expected or not.

Yes, both two are expected. I didn't see anything wrong in the log file...

The first one is related on how carla is run. After multiple tries, the only way that worked for me is: /CarlaUE4.sh -quality-level=Epic -prefernvidia -world-port=2000 -RenderOffScreen. Multiple attempts failed because I did not provide -prefernvidia

Thank you for the useful information!

The second issue is due to not using a computer powerful enough to spawn multiple vehicles. In my case I start rviz for the first vehicle but not for the second vehicle. I initialize the vehicles and send goals to the vehicles by writing my own node and I do not use any of the RViz interface (e.g., 2D pose estimate).

I'm busy with refactoring the bridge for sensors these days, and it works well in humble version. Now I'm verifying it in galactic version. Maybe you can try again later.

Before I close this ticket, do you mind telling me if it's possible to spawn more than two vehicles if I want to? Is your package limited to only two vehicles?

It's possible in theory, but I haven't tried that before. Feel free to give me feedback if you try that.

[evshary]

@zeidk If you are still trying, could you please try this way? Originally you run ros2 launch autoware_carla_launch autoware_carla.launch.xml becomes Run these two commands first poetry run python3 main.py --host $(env CARLA_SIMULATOR_IP) --rolename 'v1' --position 87.687683,145.671295,0.300000,0.000000,90.000053,0.000000" cwd="$(env AUTOWARE_CARLA_ROOT)/external/zenoh_carla_bridge/carla_agent poetry run python3 main.py --host $(env CARLA_SIMULATOR_IP) --rolename 'v2' --position 92.109985,227.220001,0.300000,0.000000,-90.000298,0.000000" cwd="$(env AUTOWARE_CARLA_ROOT)/external/zenoh_carla_bridge/carla_agent then $(env AUTOWARE_CARLA_ROOT)/external/zenoh_carla_bridge/target/release/zenoh_carla_bridge

I put these in the launch files before, but it seems like the order matters. Not sure whether this is the issue you met.

Note that you might need to open 3 terminal with source env.sh before running these 3 commands separately.

[zeidk]

Hello, @evshary it seems that starting the bridge last gives better outcome. Before, autoware will get stuck in the "initializing" state but now it does not anymore. This works perfectly fine for v1.

The current issue is that when starting the second vehicle I have some errors with Carla. As soon as I spawn the second vehicle with:

poetry run python3 main.py --host $(env CARLA_SIMULATOR_IP) --rolename 'v2' --position 92.109985,227.220001,0.300000,0.000000,-90.000298,0.000000" cwd="$(env AUTOWARE_CARLA_ROOT)/external/zenoh_carla_bridge/carla_agent

I get:

4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
MESA-INTEL: warning: Performance support disabled, consider sysctl dev.i915.perf_stream_paranoid=0

ERROR: Invalid session: no stream available with id 2 
ERROR: Invalid session: no stream available with id 4 
ERROR: Invalid session: no stream available with id 5 
ERROR: Invalid session: no stream available with id 6 
...

I briefly read that could be an issue with sensors not being deleted before sensors for vs are spawned. Any experience with this? Should I modify loop.py?

[evshary]

I've updated the script to make sure bridge runs later than python spawn script. For your issue, yes, I sometimes saw that, but I haven't dived deep into it because it doesn't affect the scenario. If you found anything interesting, we can have further discussion.

[zeidk]

@evshary Thank you. I will give it a try today and I will report back. Thank you for taking the time to work on this.

[zeidk]

@evshary I would like to express my highest gratitude for your help. Your solution for starting the bridge last fixed all the issues. I even managed to spawn and move 3 vehicles (it's a bit slow). We will have an AV workshop in September and it's free. Please attend if you can, we will have interesting keynote speakers. I will use your package to demonstrate a demo.

https://www.nist.gov/news-events/events/2023/09/standards-and-performance-metrics-road-automated-vehicles-workshop

Thank you