search

evshary / autoware_carla_launch

ROS 2 Launch for running Autoware + Carla easily
https://autoware-carla-launch.readthedocs.io/en/latest/
Apache License 2.0
62 stars 19 forks source link

How does Carla's time management affect Autoware's timers? #36

Open MarioAlmelaUMH opened 1 year ago

MarioAlmelaUMH commented 1 year ago

Hello, I was trying to monitor different Autoware topics, belonging to different modules:

/sensing/camera/traffic_light/image_raw, /sensing/imu/tamagawa/imu_raw, /sensing/lidar/concatenated/pointcloud, /sensing/gnss/pose_with_covariance, /perception/object_recognition/detection/clustering/clusters, /perception/object_recognition/detection/clustering/objects, /perception/object_recognition/detection/objects, /perception/object_recognition/objects, /system/component_state_monitor/component/autonomous/perception, /control/command/control_cmd, /perception/object_recognition/detection/rois0, perception/object_recognition/detection/camera_lidar_fusion/objects

I was recording them in a ros2bag and then checking both the timestamp of the messages' header as well as the ros2bag recording timestamp, as well as the order in which these messages are being recorded. I've noticed some strange behavior, however. As I see in these links: https://github.com/autowarefoundation/autoware.universe/blob/main/control/trajectory_follower_node/src/simple_trajectory_follower.cpp, https://github.com/autowarefoundation/autoware.universe/blob/main/control/shift_decider/src/shift_decider.cpp, Autoware's control modules are set on a timer, in some cases with frequencies around 30 ms. However Carla's default settings set the delta_seconds to 50 ms (delta_time=0.05).

Additionally, it is difficult for the whole platform of Carla, Zenoh bridge and Autoware to work at perfect real time, so the Carla ticks not always tick with a true 50 ms frequency (a tick can sometimes last for a real time second, or half a second, for example). That being said, when monitoring the control topics working on a 30 ms timer, I'd expect to see a bunch of control messages being recorded within each Carla tick, but that doesn't happen. Only one control message appears within each Carla tick.

I've added a CSV screenshot containing the messages of a ros2bag I recorded while using Zenoh bridge. Not all the topics I've listed above were recorded properly, but I hope it serves to illustrate some of the behavior I'm talking about. This was recorded when the ego vehicle was moving normally on the road. It's not edited in any way. I've used this: https://ternaris.gitlab.io/rosbags/index.html to convert the ros2bag to CSV. The E column called timestamp refers to the timestamp stored in each message's header, while the G column called bag_timestamp refers to the timestamp when each message was recorded into the ros2bag.

trace

Given all this:

Sorry for the avalanche of questions, but understanding how Carla's time management and Autoware's relate to each other is critical for the project I'm working on.

Thank you so much for your answers.

evshary commented 1 year ago

Since I provide the /clock and set use_sim_time to true in Autoware, Autoware should use the timestamp in /clock. That is ROS Time.

For Carla ticker, I tick it in the independent thread here. And everytime it ticks, it also published /clock topic. In my setting, 1 Carla simulated second = 2 real world seconds.

For the frequency of control command, it's an interesting topic. I didn't analyze what it should be before. But it shouldn't be related to ticking Carla.

For the ros2bag, I don't have too much knowledge on that.

MarioAlmelaUMH commented 1 year ago

Thank you for your answer. I've been trying to change the simulation base timestep of 50ms to 1 ms. I've changed this file, both in the fixed_delta_seconds=Some(0.05); as well as in the timer on the bottom (lines 143-144). I've also changed the Carla agent file where the same definition is made.

I'm running the platform in 2 different machines, one with Carla and Zenoh bridge and the other with Autoware and Autoware dds. I've made these changes in both computers.

However, when I ros2bag the Clock topic and extract it afterwards to a CSV, the timestamp is still going on 50 ms jumps, instead of the 1 ms it should be. It's possible that I'm missing some files to change, as the fixed_delta_seconds=0.05 definition was hard-coded in multiple files (and also the ROS Time during simulation was off, as it jumped in 1 ms intervals sometimes, and 50 ms intervals in some occasions too). Do you know how to effectively change simulation time step from 50 ms to 1 ms?

Thank you.

MarioAlmelaUMH commented 1 year ago

Any insight on this last question?

Thank you

evshary commented 1 year ago

Sorry for the late response. I'm quite busy these days. In fact, you don't need to change the value in Python file since we don't use sync mode in Python. I didn't use rosbag, but if I echo /clock in Autoware side, I see the clock is changed in 1ms

....
clock:
  sec: 2
  nanosec: 124546207
---
clock:
  sec: 2
  nanosec: 125546207
---
clock:
  sec: 2
  nanosec: 126546207
....
MarioAlmelaUMH commented 1 year ago

Ok, thank you for your answer. I'll double check the topic and see if the parameter change works.

MarioAlmelaUMH commented 1 year ago

Ok, I double checked it, and I'm now able to change the fixed delta seconds parameter successfully (the reason for the error was that I forgot to re-build after modifying the rust files). The ROS Time effectively moves at the correct pace now, but I'm getting these errors:

log_queue_full

Apparently, the change in message frequency is overloading the queues, making the simulation stuck. Is there a way to expand this queues so they don't get overloaded?

Thank you for your answers.

evshary commented 1 year ago

I'm not sure whether the message is related to your problem. I'm trying to adjust the timer these days, but the tick interval in Carla will affect the system and make the system not work. It needs more time to investigate. If you also find anything interesting, we can discuss it.

MarioAlmelaUMH commented 1 year ago

Ok. However I'm not a skilled developer, so I don't know if I will be able to find anything. If I find something, I'll tell you.

evshary commented 1 year ago

Do you have any reason to set the interval of Carla tick to 1ms? After some tweaking, I guess the interval smaller than 50 ms seems quite difficult to Autoware.

MarioAlmelaUMH commented 1 year ago

Yeah, I have noticed that the initial position of the vehicle becomes unstable (I try to set it up using RViz's UI, but the vehicle never lands in the right spot). Also, the simulation becomes too slow.

The reason behind this experiment was to try to synchronize the different messages that flow through Autoware and determine exactly when they are being fired. Looking through the code I noticed some nodes run on an independent timer, and we wanted a common predictable base frequency to observe when this timers fired and how they related to each other. We are also recording ROS messages in ros2bag, and the order in which the messages where being recorded seemed a bit odd, so having the base Carla world to work at 1 ms frequency could help decode this weird order.