The sensor's frame rate differ a lot on multiple GPU, version 0.9.15

lygbuaa commented 1 month ago

I am working with carla-0.9.15 on a 4*RTX3090 workstation, after the document [https://carla.readthedocs.io/en/latest/adv_multigpu/]

I start one primary server and three secondary servers, (the 4th server alwarys crash, that's another story, u can find it in another post)

Then I start the Client, with 12 cameras (7x8M, 4x3M, 1x2M), the 12 cameras are assigned to 3 GPUs automatically.

The Problem is, frame rate of the 12 cameras differs a lot, here is my log, you can see the fps of [RGBCamera][1~12] differ from 2.1 ~ 5.1 which is critical for BEV algorithms, Any Suggestions?

I24-07-25 08:11:26.159    1799 sensor_wrapper.py:524] [RGBCamera][7] frame: 1100, fps: 3.91, timestamp: 1721895086.159199, shape: (2160, 3840, 3), processing_time: 0.000375
I24-07-25 08:11:26.490    1799 sensor_wrapper.py:524] [RGBCamera][4] frame: 1100, fps: 3.99, timestamp: 1721895086.490569, shape: (2160, 3840, 3), processing_time: 0.032702
I24-07-25 08:11:36.087    1799 sensor_wrapper.py:524] [RGBCamera][10] frame: 1400, fps: 5.17, timestamp: 1721895096.087878, shape: (1560, 1920, 3), processing_time: 0.000262
I24-07-25 08:11:37.133    1799 sensor_wrapper.py:524] [RGBCamera][5] frame: 1000, fps: 3.54, timestamp: 1721895097.133593, shape: (2160, 3840, 3), processing_time: 0.000253
I24-07-25 08:11:39.514    1799 sensor_wrapper.py:524] [RGBCamera][1] frame: 600, fps: 2.10, timestamp: 1721895099.514028, shape: (2160, 3840, 3), processing_time: 0.217643
I24-07-25 08:11:39.948    1799 sensor_wrapper.py:524] [RGBCamera][3] frame: 1400, fps: 4.65, timestamp: 1721895099.948733, shape: (2160, 3840, 3), processing_time: 0.000062
I24-07-25 08:11:40.153    1799 sensor_wrapper.py:524] [RGBCamera][12] frame: 1500, fps: 4.95, timestamp: 1721895100.153684, shape: (1080, 1920, 3), processing_time: 0.000128
I24-07-25 08:11:40.540    1799 sensor_wrapper.py:524] [RGBCamera][9] frame: 1500, fps: 4.91, timestamp: 1721895100.540316, shape: (1560, 1920, 3), processing_time: 0.000111
I24-07-25 08:11:41.889    1799 sensor_wrapper.py:524] [RGBCamera][6] frame: 1500, fps: 5.07, timestamp: 1721895101.889237, shape: (2160, 3840, 3), processing_time: 0.001303
I24-07-25 08:11:45.067    1799 sensor_wrapper.py:524] [RGBCamera][8] frame: 1100, fps: 3.99, timestamp: 1721895105.067230, shape: (1560, 1920, 3), processing_time: 0.015353
I24-07-25 08:11:45.104    1799 sensor_wrapper.py:524] [RGBCamera][11] frame: 1100, fps: 4.00, timestamp: 1721895105.104382, shape: (1560, 1920, 3), processing_time: 0.000239

My host machine is ubuntu 22.04.1, nvidia-dkms-535， cuda 12.2, Vulkan Instance Version: 1.3.204， Vulkan apiVersion 1.3.242.

EmmettJZK commented 1 month ago

我理解了一下你的工况，你看我理解的对吗：12个Camera分配到三个Secondary上，并且在Python端统计FPS。首先我可以分享几个主从架构的信息： 1、在同一个Secondary CARLA上的传感器，World FPS是一致的。 2、Python端是通过RPC/TCP传输，LibCarla反序列化，Callback函数处理后的结果。在Python Client端统计FPS会很不准确。我们已实现11V3R1L的实时方案，感兴趣的话可以联系我们深信科创。我的微信号：1824729026

I have reviewed your situation and want to make sure I understand it correctly: 12 cameras are assigned to three secondary CARLA, and FPS is being calculated on the Python client side. First, let me share some information about the primary-secondary architecture: 1.The World FPS for sensors on the same Secondary CARLA is consistent. 2.On the Python client side, the data is transmitted via RPC/TCP, deserialized by LibCarla, and processed by callback functions. Calculating FPS on the Python client side can be quite inaccurate.

PatrickPromitzer commented 1 month ago

Hi, if it is important to have a constant fps, use the synchronization mode. https://carla.readthedocs.io/en/latest/foundations/#setting-synchronous-mode

Example: https://github.com/carla-simulator/carla/blob/dev/PythonAPI/examples/sensor_synchronization.py

Without the synchronization mode, the sensor data comes if Carla is done processing, which is not constant.

EmmettJZK commented 1 month ago

Hi, if it is important to have a constant fps, use the synchronization mode. https://carla.readthedocs.io/en/latest/foundations/#setting-synchronous-mode

Example: https://github.com/carla-simulator/carla/blob/dev/PythonAPI/examples/sensor_synchronization.py

Without the synchronization mode, the sensor data comes if Carla is done processing, which is not constant.

Well, actually, CARLA is set to synchronization mode if it's Muli-GPU mode or Primary-Secondary mode. But the Secondary CARLA can be Asynchronous, the frame rate of sensor depend on capability of hardwares, like CPU or GPU. See code in CarlaEngine.cpp

void FCarlaEngine::OnPreTick(UWorld *, ELevelTick TickType, float DeltaSeconds)
{
  TRACE_CPUPROFILER_EVENT_SCOPE_STR(__FUNCTION__);
  if (TickType == ELevelTick::LEVELTICK_All)
  {

    if (bIsPrimaryServer)
    {
      if (CurrentEpisode && !bSynchronousMode && SecondaryServer->HasClientsConnected()) 
      {
        // set synchronous mode
        CurrentSettings.bSynchronousMode = true;
        CurrentSettings.FixedDeltaSeconds = 1 / 20.0f;
        OnEpisodeSettingsChanged(CurrentSettings);
        CurrentEpisode->ApplySettings(CurrentSettings);
      }

      // process RPC commands
      do
      {
        Server.RunSome(1u);
      }
      while (bSynchronousMode && !Server.TickCueReceived());
    }
    else
    {
      // process frame data
      do
      {
        Server.RunSome(1u);
      }
      while (!FramesToProcess.size());
    }

    // update frame counter
    UpdateFrameCounter();

    if (CurrentEpisode)
    {
      CurrentEpisode->TickTimers(DeltaSeconds);

      if (!bIsPrimaryServer)
      {
        if (FramesToProcess.size())
        {
          TRACE_CPUPROFILER_EVENT_SCOPE_STR("FramesToProcess.PlayFrameData");
          std::lock_guard<std::mutex> Lock(FrameToProcessMutex);
          FramesToProcess.front().PlayFrameData(CurrentEpisode, MappedId);
          FramesToProcess.erase(FramesToProcess.begin()); // remove first element
        }
      }
    }
  }
}

PatrickPromitzer commented 1 month ago

I don't know where you get the fps value, but I guess you calculate it. The fps value is not in the output of the Camera Sensor https://carla.readthedocs.io/en/0.9.15/ref_sensors/#rgb-camera

The frame rate is always different from the real time, except you write a code to synchronize the time of the simulation to the real time. I can't tell if it is right, without seeing how you calculate the fps, but I noticed that the timestamp is real time and not simulation time. If you calculate the fps with real time and not simulation time, it is expected to not have a constant fps.

To be sure the camera sensor is not the problem, you can set the "sensor_tick" of the sensor to "fixed_delta_seconds" but this should not make a difference.

Did you test and look if the synchronization mode is active, or did you just look in the code and expect to be active?

client = carla.Client(HOST, PORT) 
world = client.get_world()
settings = world.get_settings()
print(settings.synchronous_mode)

carla-simulator / carla

The sensor's frame rate differ a lot on multiple GPU, version 0.9.15 #7986