Question about inference speed

[adeebislam8]

Hi,

I've been running some experiments and noticed something curious. While I measured a model inference speed of around 0.05 seconds on an RTX 4080, there’s an additional latency of about 0.05 seconds during the CARLA world tick. I suspect this is because of the fixed time step set for the CARLA world.

When I increase the framerate of the CARLA simulation to more than 20Hz in leaderboard_evaluator_local.py, the evaluation fails with an error message saying,

"RuntimeError: A sensor took too long to send their data."

This makes me think that all sensors need to send their measurements within 0.05 seconds. I’ve tried adjusting the sensor reading frequency, but that didn’t help.

I’m curious why this requirement exists and would appreciate any insights. Also, I’d love to know if there are any ways to reduce the time taken in the tick function.

Thanks for your help!

# in scenario_manager_local.py -> _tick_scenario() function

        time1 = time.time()
        if self._running and self.get_running_status():
            CarlaDataProvider.get_world().tick(self._timeout)
        time2 = time.time()
        print("Time tick world: ", time2 - time1)

[Kait0]

If you intend to measure the performance of TF++ (or any leaderboard model) you need to measure the time of the run_step function. This is also how the measurement in Table 8 was done. It makes sense to exclude the time of the simulation stack because on a real car you don't have to simulate anything.

As for your particular problem, that the CARLA tick happens to use 0.05 real seconds on your setup is likely coincidence. If you use a different CARLA town or a different traffic density you will likely observe different results. Simulation time and real time are decoupled in sychronous mode (which CARLA is set to in the leaderboard). The simulated time will always be 0.05 seconds exactly, the real time can vary. You can find more information about this in the CARLA documentation.

I am not aware of a way to speed up the CARLA simulator (without sacrificing something) otherwise I would have already implemented it. You can speed up the simulation if you reduce the number of background traffic agents (but this will make the problem easier), because the traffic manager in CARLA is relatively slow.

[adeebislam8]

Got it! Thanks a lot for the quick explanation and the amazing repo. Learning a lot!