Why the images from camera are not continuous?

[huangchunan]

CARLA version:CARLA 0.9.15 Platform/OS:ubuntu 20.04 Questions ： I attached a camera to my vehicle. But the generated images are not continuous, like from 00082670.png to 00082761.png Any answer is welcome！

def __init__(self, parent_actor, hud, gamma_correction):
    self.sensor = None
    self.surface = None
    self._parent = parent_actor
    self.hud = hud
    self.recording = False
    bound_x = 0.5 + self._parent.bounding_box.extent.x
    bound_y = 0.5 + self._parent.bounding_box.extent.y
    bound_z = 0.5 + self._parent.bounding_box.extent.z
    Attachment = carla.AttachmentType

    self._camera_transforms = [(carla.Transform(carla.Location(x=0.1419, y=0, z=1.649)), Attachment.Rigid)]

    self.transform_index = 0
    self.sensors = [['sensor.camera.rgb', cc.Raw, 'Camera RGB', {}]]
    world = self._parent.get_world()
    bp_library = world.get_blueprint_library()
    bp = bp_library.find(self.sensors[0][0])
    if self.sensors[0][0].startswith('sensor.camera'):
        bp.set_attribute('image_size_x', str(hud.dim[0]))
        bp.set_attribute('image_size_y', str(hud.dim[1]))
    if bp.has_attribute('gamma'):
        bp.set_attribute('gamma', str(gamma_correction))
    for attr_name, attr_value in self.sensors[0][3].items():
        bp.set_attribute(attr_name, attr_value)
    bp.set_attribute('sensor_tick', str(0.1))
    self.sensors[0].append(bp)
    self.index = None

def set_sensor(self, index, notify=True, force_respawn=False):
    index = 0
    needs_respawn = True if self.index is None else \
        (force_respawn or (self.sensors[index][2] != self.sensors[self.index][2]))
    if needs_respawn:
        if self.sensor is not None:
            self.sensor.destroy()
            self.surface = None
        self.sensor = self._parent.get_world().spawn_actor(
            self.sensors[index][-1],
            self._camera_transforms[self.transform_index][0],
            attach_to=self._parent,
            attachment_type=self._camera_transforms[self.transform_index][1])
        # We need to pass the lambda a weak reference to self to avoid
        # circular reference.
        weak_self = weakref.ref(self)
        self.sensor.listen(lambda image: Camera_h30._parse_image(weak_self, image))
    if notify:
        self.hud.notification(self.sensors[index][2])
    self.index = index

def render(self, display):
    if self.surface is not None:
        display.blit(self.surface, (0, 0))

@staticmethod
def _parse_image(weak_self, image):
    self = weak_self()
    if not self:
        return
    image.convert(self.sensors[self.index][1])
    array = np.frombuffer(image.raw_data, dtype=np.dtype("uint8"))
    array = np.reshape(array, (image.height, image.width, 4))
    array = array[:, :, :3]
    array = array[:, :, ::-1]
    self.surface = pygame.surfarray.make_surface(array.swapaxes(0, 1))
    output_path= "/tmp/camera_h30"
    image.save_to_disk(os.path.join(output_path, '%08d' % image.frame))

[jorge-kabuto]

To guarantee the continuity of images, you should instead run the client is synchronous mode. There is a working example of this in "examples\synchronous_mode.py". This will ask the server to wait until the client recieves a frame, and only continues the simulation when recieving confirmation.

[huangchunan]

To guarantee the continuity of images, you should instead run the client is synchronous mode. There is a working example of this in "examples\synchronous_mode.py". This will ask the server to wait until the client recieves a frame, and only continues the simulation when recieving confirmation.

I have used 'python3 manual_control.py --sync' to guarantee the continuity. Does it not work? It seems to lost some images. For example, I ran the simulation for 15 seconds, but it only generated the first 5 seconds images.

[Blyron]

It should guarantee... We will look at that

[joel-mb]

@huangchunan The manual_control script does not guarantee the continuity of the data, even in synchronous mode.

To guarantee the continuity of the data, besides running the server in synchronous mode, you need to implement a sensor barrier. The sensor barrier is a client side mechanism that waits until all data has been received and processed to tick again the server. This mechanism is implemented in all our scripts that need to guarantee sensor continuity for all frames (e.g., leaderboard).

Please take a look to this script that introduces this mechanism: https://github.com/carla-simulator/carla/blob/dev/PythonAPI/examples/sensor_synchronization.py

[huangchunan]

@huangchunan The manual_control script does not guarantee the continuity of the data, even in synchronous mode.

To guarantee the continuity of the data, besides running the server in synchronous mode, you need to implement a sensor barrier. The sensor barrier is a client side mechanism that waits until all data has been received and processed to tick again the server. This mechanism is implemented in all our scripts that need to guarantee sensor continuity for all frames (e.g., leaderboard).

Please take a look to this script that introduces this mechanism: https://github.com/carla-simulator/carla/blob/dev/PythonAPI/examples/sensor_synchronization.py

@joel-mb I tried and it worked. However, the fps is very low, about 2 fps. I found the core problem is the speed of generating images is far too slow than the simulation speed. Anyway to accelerate the speed? Maybe The manual_control script could guarantee the continuity of the data in synchronous mode, but the speed of generating images is too slow.

[joel-mb]

@huangchunan Performance is highly dependent on your hardware and camera configuration (e.g., number of cameras, resolution, ...). Also, save_to_disk is known to be a very slow operation, so maybe this is the cause of your problem.

How many cameras are you spawning? Could you share the script you are using?

[huangchunan]

@huangchunan Performance is highly dependent on your hardware and camera configuration (e.g., number of cameras, resolution, ...). Also, save_to_disk is known to be a very slow operation, so maybe this is the cause of your problem.

How many cameras are you spawning? Could you share the script you are using?

Hardware: CPU (i9-9900k), GPU(GeForce RTX 2080Ti) I added three cameras in manual_control.py. Any function to replace save_to_disk ?

class Camera_h30(object):

def __init__(self, parent_actor, hud, gamma_correction, sensor_queue, sensor_list):
    self.sensor = None
    self.surface = None
    self._parent = parent_actor
    self.hud = hud
    self.recording = False
    Attachment = carla.AttachmentType

    self._camera_transforms = [(carla.Transform(carla.Location(x=0.1419, y=0, z=1.649)), Attachment.Rigid)]

    self.transform_index = 0
    self.sensors = [['sensor.camera.rgb', cc.Raw, 'Camera RGB', {}]]
    world = self._parent.get_world()
    bp_library = world.get_blueprint_library()
    bp = bp_library.find(self.sensors[0][0])
    if self.sensors[0][0].startswith('sensor.camera'):
        bp.set_attribute('image_size_x', str(1920))
        bp.set_attribute('image_size_y', str(1080))
    if bp.has_attribute('gamma'):
        bp.set_attribute('gamma', str(gamma_correction))
    for attr_name, attr_value in self.sensors[0][3].items():
        bp.set_attribute(attr_name, attr_value)
    # bp.set_attribute('sensor_tick', str(0.1))
    bp.set_attribute('fov', str(28))
    self.sensors[0].append(bp)
    self.index = None
    index = 0
    force_respawn=False
    needs_respawn = True if self.index is None else \
        (force_respawn or (self.sensors[index][2] != self.sensors[self.index][2]))
    if needs_respawn:
        if self.sensor is not None:
            self.sensor.destroy()
            self.surface = None
        self.sensor = self._parent.get_world().spawn_actor(
            self.sensors[index][-1],
            self._camera_transforms[self.transform_index][0],
            attach_to=self._parent,
            attachment_type=self._camera_transforms[self.transform_index][1])

        output_path= "/tmp/camera_h30"
        self.sensor.listen(lambda image: sensor_callback(image, sensor_queue, "camera_h30", output_path))
        sensor_list.append(self.sensor)

    self.hud.notification(self.sensors[index][2])
    self.index = index

def sensor_callback(sensor_data, sensor_queue, sensor_name, output_path):
    # Do stuff with the sensor_data data like save it to disk
    # Then you just need to add to the queue
    sensor_data.save_to_disk(os.path.join(output_path, '%06d' % sensor_data.frame))
    sensor_queue.put((sensor_data.frame, sensor_name))

[joel-mb]

@huangchunan From what I understand you are adding these three cameras to the manual_control script. I would suggest creating you own script for this purpose. manual_control could be also a bottleneck due to data transformation and the usage of pygame.

You should get some extra fps moving to a new script. However, there is no alternative to save_to_disk that is known to be slow.