[Strange Performance] Unusual Fluctuations in Local Trajectories while Running Apollo in Carla

[AOOOOOA]

Hi,

I'm running Apollo(0.9.8) in Carla(0.9.14) using the official bridge provided by guardstrikelab (GitHub link). I attempted to simulate a realistic driving scenario by adding multiple vehicles in Carla alongside Apollo, using the generate_traffic.py script from the PythonAPI/examples folder. Specifically, I added 20 vehicles and 20 walkers, setting them to safe mode to avoid collisions. Carla was configured in synchronous mode with a frame time of 0.05.

However, after adding the additional vehicles and walkers, I noticed unusual fluctuations in Apollo's local trajectory. This issue rarely occurs when running Apollo alone in Carla. Most of the time, Apollo manages to correct itself and return to the proper path, but occasionally it gets stuck.

Is this behavior expected?

I've attached a sample video below for reference.

https://github.com/user-attachments/assets/80540900-b938-4f17-91d0-29f6c5124fba

[PatrickPromitzer]

Hi, I can't say how Apollo works with Carla, but try 0.1 for the frame time.

With more Actors, sensors, and objects (in the map), the simulation is getting slower. My guess is, Apollo is getting confused if a frame is missing or to late.

[AOOOOOA]

Hi, I can't say how Apollo works with Carla, but try 0.1 for the frame time.

With more Actors, sensors, and objects (in the map), the simulation is getting slower. My guess is, Apollo is getting confused if a frame is missing or to late.

Yes I got your point.

After investigating the issue further, I discovered that the root cause of the Apollo vehicle getting stuck is that Carla becomes too busy handling other actors, sensors, and tasks. As a result, it fails to process the control messages from the Apollo control topic in a timely manner. Apollo uses CyberRT (a ROS-like framework) for message passing, where topics use shared memory for communication. When Carla is overloaded, it cannot retrieve control messages from the shared memory fast enough, leading to a read buffer overflow.

Moreover, since Carla does not have a control command timestamp validation mechanism, it ends up executing control commands that have a significant delay from the last processed command, causing errors in the vehicle's behavior.

Currently, I am only running a single Apollo agent in Carla, but if multiple ADS systems were connected simultaneously, this issue would likely become more severe. How is this situation typically handled? Could this be considered a bug in the system? Additionally, has there been any consideration of implementing a timestamp validation mechanism on the Carla side to prevent such delays and ensure control commands are executed in a timely manner?

[PatrickPromitzer]

The common way is using the Synchronous mode. In this mode Carla is waiting at each step until it gets the permission to run the next step. For example: 1.) Carla is waiting 2.) code is running 3.) you send a command to give Carla the permission to tick 4.) Carla is running for 0.1 seconds 5.) repeat 1-4

There are examples for this process, and even if you can't get it to real time, you have your component synchronized. With this mode, it doesn't care if the sensor take 0.001 seconds or 10.0 seconds, because you make Carla wait until you did everything you need it t do. https://carla.readthedocs.io/en/0.9.15/adv_traffic_manager/#synchronous-mode https://github.com/carla-simulator/carla/blob/ue4-dev/PythonAPI/examples/sensor_synchronization.py

I didn't work with the ROS part of Carla, but maybe there is a synchronous-mode for the ROS side, too. It looks less like a bug, and more like very simple solution to make it work. I can't say if it is a good or bad solution because I didn't try it. If the ROS part doesn't support the synchronous-mode (It would be strange if it isn't supported, because it should be possible), it would be better for you to have an python code as an interface between the Apollo and Carla instance.

[AOOOOOA]

The common way is using the Synchronous mode. In this mode Carla is waiting at each step until it gets the permission to run the next step. For example: 1.) Carla is waiting 2.) code is running 3.) you send a command to give Carla the permission to tick 4.) Carla is running for 0.1 seconds 5.) repeat 1-4

There are examples for this process, and even if you can't get it to real time, you have your component synchronized. With this mode, it doesn't care if the sensor take 0.001 seconds or 10.0 seconds, because you make Carla wait until you did everything you need it t do. https://carla.readthedocs.io/en/0.9.15/adv_traffic_manager/#synchronous-mode https://github.com/carla-simulator/carla/blob/ue4-dev/PythonAPI/examples/sensor_synchronization.py

I didn't work with the ROS part of Carla, but maybe there is a synchronous-mode for the ROS side, too. It looks less like a bug, and more like very simple solution to make it work. I can't say if it is a good or bad solution because I didn't try it. If the ROS part doesn't support the synchronous-mode (It would be strange if it isn't supported, because it should be possible), it would be better for you to have an python code as an interface between the Apollo and Carla instance.

I've already set Carla to run in synchronous mode from the start, with a default step time of 0.05 seconds. Apollo and Carla communicate via CyberRT, which is a ROS-like framework. In this setup, Apollo publishes control commands, and Carla is supposed to receive and execute them.

Since Apollo's control module updates at a high frequency (50 Hz in this case), it continuously sends control commands to Carla. In the Carla-Apollo-Bridge, there's a callback function that gets triggered every time Apollo publishes a control command. This function should then call vehicle.apply_control() with the updated command. According to the Carla documentation, the simulator will execute the last control command it receives before the next simulation tick.

The issue arises when multiple NPC actors are generated in Carla's world. When this happens, Carla becomes too busy to handle the callback function in a timely manner. CyberRT, which uses shared pointers to access the control command data, can get overloaded if Carla doesn't process the commands quickly enough. This leads to the data buffer in the CyberRT topic filling up, causing older control commands to be dropped. Over time, this accumulation of missed commands results in the Apollo vehicle behaving incorrectly.

I've noticed that there doesn't seem to be a timestamp checking mechanism in Carla, which makes it difficult for Carla to determine whether the control data it receives is the most up-to-date. I wonder if this could be contributing to the issue.

I'm also curious whether such a mechanism also exists in the traffic manager module, or if there are other important pieces of information that might also suffer from timeliness issues. If that's the case, it could lead to various performance problems, particularly when the system is handling a high workload.

[PatrickPromitzer]

In a normal case this should now be a problem.

With world.tick() Carla is updating the simulation, and if you want to get Actor data, like the position of every vehicle, it should be up to date after calling the world.tick() function.

There a 2 types of sensors.

• sending data each step
• sending data after an event

I only found 2 sensors which doesn't send data each step.

• sensor.other.obstacle - sending data if an Actor was detected
• sensor.other.lane_invasion - sending data if a lane was crossed

Every other sensor will send data to the python API (probably ROS, too) each step. With this information, you can use a Queue and wait until the data arrived.

from queue import Queue

sensor_transform = carla.Transform(
    carla.Location(
        x=0.0,
        y=0.0,
        z=3.0),
    carla.Rotation(
        pitch=0.0,
        yaw=0.0,
        roll=0.0))

# connect to server
client.set_timeout(2.0)](client: carla.Client = carla.Client("127.0.0.1", 2000))

# set Carla to sync mode
settings = world.get_settings()
settings.synchronous_mode = True
settings.fixed_delta_seconds = 0.1
world.apply_settings(settings)

# get camaera sensor
world: carla.World = client.get_world()
bp_lib = world.get_blueprint_library()
cam_sensor_bp = bp_lib.find("sensor.camera.rgb")

#spawn camera sensor
sensor_cam = world.try_spawn_actor(cam_sensor_bp, sensor_transform)

#add camera listener
sensor_queue = Queue()
sensor_cam.listen( lambda item: sensor_queue.put(item)) # putting the data in the queue if Carla send one

while True:
    # tell Carla to run for a step
    world.tick()

    # waiting for the date to arrive
    sensor_data =  sensor_queue.get() 

    # this point is only reached if the data of the sensor has arrived 

With this setup you have 2 things.

• The script is telling Carla when it should run "world.tick()"
• The script waits for the sensor data of Carla and doesn't continue if no sensor data reached the script "sensor_queue.get()"

The only problem which could happen is, if more than one Client is using the "world.tick()" function. If more than one client is using the "world.tick()" function, you lose the synchronous aspect, because both clients think they control the timing of Carla, and two ticks happen while only one should.

I have 2 guesses in your situation. 1.) one Client tries to keep the simulation real time, and because of that reason it skip frames 2.) Two clients try to use "world.tick()" and because of that reason Carla is running as fast as possible, which make it lose the synchronous aspect

In both cases it can get to the issue you have, but it could be something else I didn't thought about, too.

[AOOOOOA]

In a normal case this should now be a problem.

With world.tick() Carla is updating the simulation, and if you want to get Actor data, like the position of every vehicle, it should be up to date after calling the world.tick() function.

There a 2 types of sensors.

• sending data each step
• sending data after an event

I only found 2 sensors which doesn't send data each step.

• sensor.other.obstacle - sending data if an Actor was detected
• sensor.other.lane_invasion - sending data if a lane was crossed

Every other sensor will send data to the python API (probably ROS, too) each step. With this information, you can use a Queue and wait until the data arrived.

from queue import Queue

sensor_transform = carla.Transform(
    carla.Location(
        x=0.0,
        y=0.0,
        z=3.0),
    carla.Rotation(
        pitch=0.0,
        yaw=0.0,
        roll=0.0))

# connect to server
client.set_timeout(2.0)](client: carla.Client = carla.Client("127.0.0.1", 2000))

# set Carla to sync mode
settings = world.get_settings()
settings.synchronous_mode = True
settings.fixed_delta_seconds = 0.1
world.apply_settings(settings)

# get camaera sensor
world: carla.World = client.get_world()
bp_lib = world.get_blueprint_library()
cam_sensor_bp = bp_lib.find("sensor.camera.rgb")

#spawn camera sensor
sensor_cam = world.try_spawn_actor(cam_sensor_bp, sensor_transform)

#add camera listener
sensor_queue = Queue()
sensor_cam.listen( lambda item: sensor_queue.put(item)) # putting the data in the queue if Carla send one

while True:
    # tell Carla to run for a step
    world.tick()

    # waiting for the date to arrive
    sensor_data =  sensor_queue.get() 

    # this point is only reached if the data of the sensor has arrived 

With this setup you have 2 things.

• The script is telling Carla when it should run "world.tick()"
• The script waits for the sensor data of Carla and doesn't continue if no sensor data reached the script "sensor_queue.get()"

The only problem which could happen is, if more than one Client is using the "world.tick()" function. If more than one client is using the "world.tick()" function, you lose the synchronous aspect, because both clients think they control the timing of Carla, and two ticks happen while only one should.

I have 2 guesses in your situation. 1.) one Client tries to keep the simulation real time, and because of that reason it skip frames 2.) Two clients try to use "world.tick()" and because of that reason Carla is running as fast as possible, which make it lose the synchronous aspect

In both cases it can get to the issue you have, but it could be something else I didn't thought about, too.

1. I’m carefully managing the synchronization process. I generate multiple NPCs using generate_traffic.py through a separate client, but I rely solely on the Carla_Apollo_bridge client to control synchronization via world.tick(). In generate_traffic.py, I modified the command to world.wait_for_tick() to prevent the scenario where two clients attempt to tick the world simultaneously. Therefore, I don’t believe this is the cause of the issue.
2. There seems to be some misunderstanding. In the example code you provided, the world appears to be ticked only when sensor data arrives. However, in my case, the issue stems from a large volume of control data continuously coming from Apollo in the whole simulation process, which Carla needs to process. The update frequency of this control data is higher than Carla's simulation frequency. While it's expected that Carla may not handle every single piece of control data, as the ADS control module sends data at a very high frequency, missing too much control data is undesirable. Even when I use a queue after world.tick(), the issue persists because Carla becomes overwhelmed by other tasks and doesn’t have "enough" time to process all the incoming control data. As a result, older control data continues to be dropped from the queue.

I tried increasing Carla's simulation frequency, but this caused the system to behave erratically. Conversely, when I decreased the simulation frequency, it led to a new problem where control commands were over-executed, an issue that has been reported by other users as well.

It seems that this issue could be resolved if Carla had an extra message-checking mechanism for handling large-scale actors and high-frequency client communication. At this point, I’m inclined to consider this a potential bug. Would you need a more specific description or code to reproduce it?

[PatrickPromitzer]

I would need to set everything up with Apollo to test it myself (currently I have too less time for that). I am only using Carla and not with the development team, but still try to find a solution to some problems.

I noticed with the python API, if you set the position to an Actor, it will get an timeout error if the server doesn't answer. I believe it would be the same for updating the control data of the Actors.

I never tested a synchronization loop without sensor data, but if the world.tick() doesn't wait until the Carla server is done with updating the requests and simulation step, it sounds like bug. I can make a short script to test it, but I can't say when I have time for it.

I don't know how the Apollo bridge works (probably ROS like), but if you can make the bridge wait for a tick, as a quick fix I suggest adding a 3th client and let this one tick very slow. It is not a "solution", but you may get at least some data (slowly).

[AOOOOOA]

I would need to set everything up with Apollo to test it myself (currently I have too less time for that). I am only using Carla and not with the development team, but still try to find a solution to some problems.

I noticed with the python API, if you set the position to an Actor, it will get an timeout error if the server doesn't answer. I believe it would be the same for updating the control data of the Actors.

I never tested a synchronization loop without sensor data, but if the world.tick() doesn't wait until the Carla server is done with updating the requests and simulation step, it sounds like bug. I can make a short script to test it, but I can't say when I have time for it.

I don't know how the Apollo bridge works (probably ROS like), but if you can make the bridge wait for a tick, as a quick fix I suggest adding a 3th client and let this one tick very slow. It is not a "solution", but you may get at least some data (slowly).

Thank you for your advice! I initially thought you were part of the development team due to your active presence in the Carla community. I'll explore some potential solutions to address the issue in the meantime, and we can also wait for the development team to confirm whether it's a bug. Once again, thank you so much for your help!

[Blyron]

We are trying to answer as much issues as possible, Co-Simulation related ones takes much time and we are on other stuff so we cannot have enough time to research this now days. We are gonna try to look at this. Try to ask at Apollo's support

[PatrickPromitzer]

I wrote a small script to test if Carla has a problem with the world.tick() update and timing. I used the python Carla package.

def test_carla_steping(server_address):

    start_position = {
        "x": 100.0,
        "y": -40.0,
        "z": 0.5
    }
    cars_amount = 25
    step_length = 28
    x_offset = 10.0
    y_offset = 0.0

    x_move = 0.0
    y_move = -10.0
    max_distance = 1.0

    client = carla.Client(server_address, 2000)
    world = client.get_world()
    original_world_settings = world.get_settings()
    original_world_settings.synchronous_mode = False

    world_settings = world.get_settings()
    world_settings.fixed_delta_seconds = 0.1
    world_settings.synchronous_mode = True
    world.apply_settings(world_settings)
    world.tick()
    blueprint_lib = world.get_blueprint_library()
    vhicle_bb = blueprint_lib.filter("vehicle.*")[0]

    vehicle_list = []
    for i in range(cars_amount):
        carla_loctaion = carla.Location(
            x=start_position["x"] + (i * x_offset),
            y=start_position["y"] + (i * y_offset),
            z=start_position["z"]
        )
        vehicle_actor = world.spawn_actor(vhicle_bb, carla.Transform(carla_loctaion))
        vehicle_list.append(vehicle_actor)
    error_list = []
    for i in range(step_length):
        world.tick()
        for index, vehicle_actor in enumerate(vehicle_list):
            location = vehicle_actor.get_location()
            position_is_ok = True
            distance_x = location.x - (start_position["x"] + (index * x_offset) + (i * x_move))
            distance_y = location.y - (start_position["y"] + (index * y_offset) + (i * y_move))
            if distance_x > max_distance or distance_x < -max_distance:
                position_is_ok = False
            if distance_y > max_distance or distance_y < -max_distance:
                position_is_ok = False
            if position_is_ok:
                error_list.append(0)
            else:
                error_list.append(1)
            location.x = start_position["x"] + (index * x_offset) + ((i + 1) * x_move)
            location.y = start_position["y"] + (index * y_offset) + ((i + 1)  * y_move)
            vehicle_actor.set_location(location)

    print(sum(error_list))

    for vehicle_actor in vehicle_list:
        vehicle_actor.destroy()
    world.apply_settings(original_world_settings)

If you want to run this script yourself, you need an empty place on a Map.

I got 0 errors if I keep the max_distance at 1.0, but if I get lower than 0.01, I start getting offsets. They probably come from the Vehicles rolling away. I tested it with world_settings.fixed_delta_seconds = 0.001 and still have no errors.

[AOOOOOA]

We are trying to answer as much issues as possible, Co-Simulation related ones takes much time and we are on other stuff so we cannot have enough time to research this now days. We are gonna try to look at this. Try to ask at Apollo's support

Sure, I will report the bug to Apollo and Carla-Apollo-Bridge. However, based on our findings, the issue seems to stem from Carla's lack of a precise timestamp determination mechanism. I would really appreciate it if you could take a closer look at this, as it appears to be a critical aspect affecting the integration.

[Blyron]

You can use simulation time as time stamp as our recorder/replayer does. So we have a way to determine the time stamps. I cannot help you more there

[AOOOOOA]

You can use simulation time as time stamp as our recorder/replayer does. So we have a way to determine the time stamps. I cannot help you more there

What I’m trying to convey is that Carla currently lacks a built-in mechanism to ensure that the control data it receives is always the most up-to-date. As a result, it processes whatever data arrives, which can occasionally cause unexpected behavior in the Autonomous Driving System (ADS), particularly when Carla is under significant computational load. While I could implement a mechanism to filter outdated data myself, this issue tends to become a hidden bug when scaling up to a large number of vehicles within the Carla environment. I suspect that many users have already experienced this issue without realizing it, and while it may seem subtle, it could lead to performance inconsistencies that would benefit from further consideration.

[Blyron]

We will review this in the future. Right now we are dealing with UE5 migration. Cannot estimate an ETA for this. If someone implements this; We are open to contributions.

[AOOOOOA]

We will review this in the future. Right now we are dealing with UE5 migration. Cannot estimate an ETA for this. If someone implements this; We are open to contributions.

I understand that the team is currently focused on the UE5 migration, and I really appreciate all the hard work going into that. That being said, I believe this joint simulation issue is quite important and worth addressing. Simulating multiple vehicles is a common use case for both Carla and Apollo, and this issue directly impacts the ability to scale simulations for realistic scenarios—something that’s essential for many users, including myself.

I’ve also noticed that a few others in the community have reported similar problems, so it seems this might not be an isolated case. If the issue continues, it could limit the usability of Carla-Apollo joint simulations for large-scale testing, which might affect ongoing projects or even discourage users from relying on this workflow.

I understand it’s not always easy to tell right away whether something is a bug or a configuration issue. If possible, could you suggest any specific diagnostics or settings I could try to help narrow down the root cause? I’m more than happy to provide additional logs, run further tests, or help out in any way I can to move this forward.

[Blyron]

I will pass this into the correct people to see if they are able to help you

[AOOOOOA]

I will pass this into the correct people to see if they are able to help you

Thank you so much for all your effort. I truly appreciate it.

[AOOOOOA]

I will pass this into the correct people to see if they are able to help you

Hi Blyron, any updates on this issue? Let me know if there’s anything that needs attention or assistance.

[Blyron]

No, we have contacted with the breach developers as in CARLA everything seems fine

[AOOOOOA]

No, we have contacted with the breach developers as in CARLA everything seems fine

Do you mean Synkrotron.ai, the developer of the Carla-Apollo-Bridge project (https://github.com/guardstrikelab/carla_apollo_bridge)?

I’ve already tried reaching out to them, but unfortunately haven’t received a reply yet. Have you had a chance to test this issue? Or could you possibly point me to someone I can directly contact? I suspect there might be some settings that still need to be configured.

[Blyron]

Our developers tested, and see the issue, but in CARLA end seems everything to be ok.

[Blyron]

And Yes, I mean Synkrotron, they are our partners so I have contacted them. I cannot point anyone to contact just wait issues answers on github

[AOOOOOA]

Our developers tested, and see the issue, but in CARLA end seems everything to be ok.

Yes, on the Carla side, everything seems fine. However, after running Apollo for a while with multiple vehicles in the Carla environment, some misbehavior occurs in Apollo. Upon further investigation, I found that the issue originates from Carla being too busy handling the environment vehicles, which prevents it from processing control data from the Apollo control topic in a timely manner.

As a result, there is a control data overflow in the Apollo control topic (not a traditional memory buffer overflow). The Apollo control topic drops the corresponding control data, and when Carla retrieves the control data, it only takes the latest data rather than the most appropriate ones. Since Carla lacks a timestamp checking mechanism, this discrepancy accumulates over time, leading to the observed misbehavior in Apollo.

Are you suggesting waiting for Synkrotron to respond on this issue, or should I report it directly on the Carla Apollo Bridge repository?

[Blyron]

Report directly in apollo bridge.

[AOOOOOA]

Report directly in apollo bridge.

I reported this issue to the bridge repository a month ago but haven’t received any response yet. It seems that the repository hasn’t been actively maintained recently.

[XiaoFei9704]

@AOOOOOA You can try to modify the frequency of the planning module: modules/planning/planning_base/gflags/planning_gflags.cc -> planning_loop_rate We can let CARLA limit the frequency of Apollo's control output by sending ticks to Apollo through CARLA Server, but this requires modifying Apollo. In order to adapt to the native Apollo, we later abandoned this solution.

[AOOOOOA]

@AOOOOOA You can try to modify the frequency of the planning module: modules/planning/planning_base/gflags/planning_gflags.cc -> planning_loop_rate We can let CARLA limit the frequency of Apollo's control output by sending ticks to Apollo through CARLA Server, but this requires modifying Apollo. In order to adapt to the native Apollo, we later abandoned this solution.

Thank you for your detailed response and for suggesting the workaround regarding modifying the frequency of the planning module. I will try this solution and see if it helps resolve the issue.

That being said, this behavior seems to cause unexpected control issues in Apollo when working with CARLA, especially under high-load scenarios with multiple vehicles in the environment. It creates noticeable performance degradation and affects the accuracy of control commands, which has a significant impact on the overall system's stability.

Would it be reasonable to classify this as a bug, or is it more of a design limitation due to the adaptation to native Apollo? I'm asking because this issue appears to deviate from expected behavior and might require further adjustments to ensure reliable performance in similar scenarios.

Once again, thank you for your support. I'll share any feedback after testing the workaround.

[yuxip]

I would consider this as a design limitation of the native Apollo, i.e., it does not allow external triggering for synchronization purpose. Take for example, when Carla is only capable of updating sensor data at 10Hz, it does not make any sense for Apollo to run planning at 50Hz (generates 4 useless plans in between sensor updates). Slowing down Apollo would be the way to go here.

I am curious as to why Apollo is able to run so fast in your experiment. How many sensors/cameras did you configure in Carla? Does your Apollo instance include the perception module or is it only running in Planning and Control mode?

[AOOOOOA]

I would consider this as a design limitation of the native Apollo, i.e., it does not allow external triggering for synchronization purposes. Take for example, when Carla is only capable of updating sensor data at 10Hz, it does not make any sense for Apollo to run planning at 50Hz (generates 4 useless plans in between sensor updates). Slowing down Apollo would be the way to go here.

I am curious as to why Apollo is able to run so fast in your experiment. How many sensors/cameras did you configure in Carla? Does your Apollo instance include the perception module or is it only running in Planning and Control mode?

I just use the default setting of Apollo 8.0, as advised in the Carla-Apollo-Bridge documentation, the simulation includes not only the Planning and Control (PnC) modules but also the Perception modules. This setup is used for co-simulation with Carla, where the vehicle state is updated by Carla.

Why the Control Module Runs Faster From my understanding , the Control Module in an ADS typically runs at a much higher frequency than other modules (e.g., Perception or Planning). This is because:

1. Vehicle dynamics (speed, direction, posture, etc.) change rapidly, requiring frequent adjustments to throttle, braking, and steering.
2. High-frequency updates allow precise trajectory tracking, stability, and safety.

To achieve this high frequency, while the other modules update at a much slower rate, multiple methods are used:

1. Trajectory Interpolation: The control module interpolates the planned trajectory to generate a denser set of waypoints, ensuring smooth and continuous updates.
2. Predicted Vehicle State: When the actual vehicle state (e.g., position, speed) updates slowly, a model-predicted state is calculated to provide a "fake" real-time input for the control module to compute control commands.
3. High-Frequency Vehicle State Updates: In real-world vehicles, the state (e.g., speed, acceleration, and position) is updated at a high frequency, which allows the control module to respond quickly to changes and adjust precisely.

Limitations in Carla-Apollo Co-Simulation In the Carla-Apollo co-simulation, the vehicle state is updated by Carla. Since Carla’s default frequency is 20Hz, the vehicle state updates are much slower than in a real-world vehicle. This creates a challenge for Apollo because the Control Module typically requires much higher update frequencies.

Apollo’s Control Module relies heavily on predicted vehicle states (calculated using a vehicle dynamics model) to generate new control commands. This dependency on predictions can introduce errors, especially in dynamic situations, as the control is not based on real-time updates. Another significant source of error arises from the issue described here: when multiple vehicles exist in the environment, Carla becomes busy handling them and fails to process control commands from Apollo in a timely manner. This delay causes the vehicle to execute outdated or unsuitable control commands once Carla resumes processing them.

Design Limitation I believe this issue reflects a limitation of the Carla-Apollo-Bridge, and more fundamentally, a limitation of Carla itself. Ideally, A driving simulator like Carla should serve the autonomous diving systems by matching the needs of them

Slowing down Apollo to match Carla’s frequency would introduce new errors and reduce the accuracy of the control module. Therefore, the problems above remain a bottleneck in co-simulation.

[yuxip]

I agree control module usually runs faster than perception and planning. But Carla's actual frame rate is simply limited by your machine's compute power. E.g., with a single machine it simply cannot keep up with the 50Hz sensor update in realtime if you have many sensor/camera installed.

However, I think you misunderstood the definition of Carla's default frequency. 20Hz, or 0.05s simulation step is in "simulation time", the actual frame rate / frequency is determined by your hardware capability. You can set simulation step to 0.02s but with a less powerful machine it can take much more than 0.02s to calculate a frame, thus the actual frame rate could be less than 50Hz which will not match Apollo's planning frequency. This is not a limitation of Carla but a limitation of compute hardware.

[yuxip]

In a Hardware-in-the-loop (HiL) testing scenario, we usually deploy Carla across multiple machines to achieve ~30Hz frame rate in real-time via distributed rendering. The vehicle dynamics model (PhysX or CarSim) will be deployed in a real-time machine to achieve 100-1000Hz compute frequency to provide more dense inputs to the ADS controller (e.g. could be running Apollo). It is only with this type of setup can we expect to work with Apollo's realtime requirement.

In a Software-in-the-loop (SiL) scenario Carla usually runs in a single machine, one cannot expect that with multi-sensor configuration Carla could match the realtime requirement of any ADS (it's simply outside the scope of SiL testing). In this scenario a proper synchronization between the simulator and the ADS is all we can ask for. Carla can be synchronized via ticks, but native Apollo (without modification) does not accept such mechanism.

[AOOOOOA]

I agree control module usually runs faster than perception and planning. But Carla's actual frame rate is simply limited by your machine's compute power. E.g., with a single machine it simply cannot keep up with the 50Hz sensor update in realtime if you have many sensor/camera installed.

However, I think you misunderstood the definition of Carla's default frequency. 20Hz, or 0.05s simulation step is in "simulation time", the actual frame rate / frequency is determined by your hardware capability. You can set simulation step to 0.02s but with a less powerful machine it can take much more than 0.02s to calculate a frame, thus the actual frame rate could be less than 50Hz which will not match Apollo's planning frequency. This is not a limitation of Carla but a limitation of compute hardware.

Does this mean that the simulation time in Carla is more of a theoretical value rather than an exact, guaranteed one? For example, if I set the time-step to 0.02s in synchronous mode, would it be correct to say that on a less powerful machine, the actual time-step might become larger than 0.02s due to computational limitations? If so, am I understanding correctly that the synchronous mode may not always maintain a perfectly consistent step length in such cases? Could you clarify how this is handled?