Crash when use 4 cameras simultaneously but okay when using 2

[LeonardMendicantBias]

Dear Carla developers, contributors, and users.

I am getting familar with Carla (version 0.9.15) to generate data. In my scenario, I attach (4) cameras on static places (i.e., on a traffic light). Since I need to obtain the exact locations for the cameras, I built Carla on Windows and played around with the UE 4 editor. I am able to place the cameras at the locations I wanted. Following the instruction from Carla, I spawn cameras and save the image via the callback function. The problem is that the simulation provide data if there are at most 2 cameras. If there are 3 cameras, the simulation starts, generate a single image for each spawned cameras, and crash with the error: "time-out of 10000ms while waiting for the simulator, make sure the simulator is ready and connected to 127.0.0.1:2000". This confused me because the error is simply not true. The simulation did generate 1 image.

I attached my code for error reproduction. It is adapted from generate_traffic.py.

import glob
import os
import sys
import time
import copy
from functools import partial

try:
    sys.path.append(glob.glob('../carla/dist/carla-*%d.%d-%s.egg' % (
        sys.version_info.major,
        sys.version_info.minor,
        'win-amd64' if os.name == 'nt' else 'linux-x86_64'))[0])
except IndexError:
    pass

import argparse
import logging
from numpy import random
import numpy as np

import carla
from carla import VehicleLightState as vls
from pascal_voc_writer import Writer

def get_actor_blueprints(world, filter, generation):
    bps = world.get_blueprint_library().filter(filter)
    if generation.lower() == "all":
        return bps
    # If the filter returns only one bp, we assume that this one needed
    # and therefore, we ignore the generation
    if len(bps) == 1:
        return bps
    try:
        int_generation = int(generation)
        # Check if generation is in available generations
        if int_generation in [1, 2, 3]:
            bps = [x for x in bps if int(x.get_attribute('generation')) == int_generation]
            return bps
        else:
            print("   Warning! Actor Generation is not valid. No actor will be spawned.")
            return []
    except:
        print("   Warning! Actor Generation is not valid. No actor will be spawned.")
        return []

def spawn_vehicles(world, client, args, synchronous_master=False):
    traffic_manager = client.get_trafficmanager(args.tm_port)
    vehicles_list = []
    blueprints = get_actor_blueprints(world, args.filterv, args.generationv)
    if not blueprints:
        raise ValueError("Couldn't find any vehicles with the specified filters")
    if args.safe:
        blueprints = [x for x in blueprints if x.get_attribute('base_type') == 'car']
    blueprints = sorted(blueprints, key=lambda bp: bp.id)
    spawn_points = world.get_map().get_spawn_points()
    number_of_spawn_points = len(spawn_points)
    if args.number_of_vehicles < number_of_spawn_points:
        random.shuffle(spawn_points)
    elif args.number_of_vehicles > number_of_spawn_points:
        msg = 'requested %d vehicles, but could only find %d spawn points'
        logging.warning(msg, args.number_of_vehicles, number_of_spawn_points)
        args.number_of_vehicles = number_of_spawn_points
    # @todo cannot import these directly.
    SpawnActor = carla.command.SpawnActor
    SetAutopilot = carla.command.SetAutopilot
    FutureActor = carla.command.FutureActor
    batch = []
    # hero = args.hero
    for n, transform in enumerate(spawn_points):
        if n >= args.number_of_vehicles:
            break
        blueprint = random.choice(blueprints)
        if blueprint.has_attribute('color'):
            color = random.choice(blueprint.get_attribute('color').recommended_values)
            blueprint.set_attribute('color', color)
        if blueprint.has_attribute('driver_id'):
            driver_id = random.choice(blueprint.get_attribute('driver_id').recommended_values)
            blueprint.set_attribute('driver_id', driver_id)
        else:
            blueprint.set_attribute('role_name', 'autopilot')
        # spawn the cars and set their autopilot and light state all together
        batch.append(SpawnActor(blueprint, transform)
            .then(SetAutopilot(FutureActor, True, traffic_manager.get_port())))
    for response in client.apply_batch_sync(batch, synchronous_master):
        if response.error:
            logging.error(response.error)
        else:
            vehicles_list.append(response.actor_id)
    # Set automatic vehicle lights update if specified
    for actor in world.get_actors(vehicles_list):
        traffic_manager.update_vehicle_lights(actor, args.car_lights_on)
    return vehicles_list

def save_image(path, key, image):
    image.save_to_disk(f'{path}/{image.frame:06d}_{key}.png')

def spawn_cameras(world, client, args, synchronous_master=False):
    camera_dict = {
        "camera-1": {
            "position": [-6270./100, 210./100, 4630./100],
            "orientation": [0, -40, 40]
        },
        "camera-2": {
            "position": [-2950./100, 3770./100, 2500./100],
            "orientation": [0, -30, -135]
        },
        "camera-3": {
            "position": [-6130./100, 3800./100, 1900./100],
            "orientation": [0, -20, -45]
        },
        "camera-4": {
            "position": [-2800./100, -200./100, 1800./100],
            "orientation": [0, -20, 135]
        }
    }
    # camera_list = []
    # for key, info in camera_dict.items():
    #     camera_bp = world.get_blueprint_library().find('sensor.camera.rgb')
    #     camera_bp.set_attribute('image_size_x', '1920')
    #     camera_bp.set_attribute('image_size_y', '1080')
    #     camera_bp.set_attribute('sensor_tick', '0.05')  # 100fps
    #     transform = carla.Transform(
    #         carla.Location(*info["position"]),
    #         carla.Rotation(info["orientation"][1], info["orientation"][2], info["orientation"][0])
    #     )
    #     camera = world.spawn_actor(camera_bp, transform)
    #     # camera.listen(lambda image: image.save_to_disk(f'{args.directory}/{key}/{image.frame:06d}.png'))
    #     camera.listen(partial(save_image, f'{args.directory}', key))
    #     camera_list.append(camera)
    #     time.sleep(0.5)
    # return camera_list
    camera_bp = world.get_blueprint_library().find('sensor.camera.rgb')
    camera_bp.set_attribute('image_size_x', '1920')
    camera_bp.set_attribute('image_size_y', '1080')
    camera_bp.set_attribute('sensor_tick', '0.05')  # 100fps
    transform_1 = carla.Transform(
        carla.Location(camera_dict["camera-1"]["position"]),
        carla.Rotation(camera_dict["camera-1"]["orientation"][1], camera_dict["camera-1"]["orientation"][2], camera_dict["camera-1"]["orientation"][0])
    )
    camera_1 = world.spawn_actor(camera_bp, transform_1)
    camera_1.listen(lambda image: image.save_to_disk(f'{args.directory}/camera-1/{image.frame:06d}.png'))
    transform_2 = carla.Transform(
        carla.Location(camera_dict["camera-2"]["position"]),
        carla.Rotation(camera_dict["camera-2"]["orientation"][1], camera_dict["camera-2"]["orientation"][2], camera_dict["camera-2"]["orientation"][0])
    )
    camera_2 = world.spawn_actor(camera_bp, transform_2)
    camera_2.listen(lambda image: image.save_to_disk(f'{args.directory}/camera-2/{image.frame:06d}.png'))
    transform_3 = carla.Transform(
        carla.Location(camera_dict["camera-3"]["position"]),
        carla.Rotation(camera_dict["camera-3"]["orientation"][1], camera_dict["camera-3"]["orientation"][2], camera_dict["camera-3"]["orientation"][0])
    )
    camera_3 = world.spawn_actor(camera_bp, transform_3)
    camera_3.listen(lambda image: image.save_to_disk(f'{args.directory}/camera-3/{image.frame:06d}.png'))
    transform_4 = carla.Transform(
        carla.Location(camera_dict["camera-4"]["position"]),
        carla.Rotation(camera_dict["camera-4"]["orientation"][1], camera_dict["camera-4"]["orientation"][2], camera_dict["camera-4"]["orientation"][0])
    )
    camera_4 = world.spawn_actor(camera_bp, transform_4)
    camera_4.listen(lambda image: image.save_to_disk(f'{args.directory}/camera-4/{image.frame:06d}.png'))
    return [camera_1, camera_2, camera_3, camera_4]

def main():
    argparser = argparse.ArgumentParser(description=__doc__)
    argparser.add_argument('--host',
        metavar='H',
        default='127.0.0.1',
        help='IP of the host server (default: 127.0.0.1)'
    )
    argparser.add_argument('-p', '--port',
        metavar='P',
        default=2000,
        type=int,
        help='TCP port to listen to (default: 2000)'
    )
    argparser.add_argument('--tm-port',
        metavar='P',
        default=8000,
        type=int,
        help='Port to communicate with TM (default: 8000)'
    )
    argparser.add_argument('--respawn',
        action='store_true',
        default=True,
        help='Automatically respawn dormant vehicles (only in large maps)'
    )
    argparser.add_argument('--filterv',
        metavar='PATTERN',
        default='vehicle.*',
        help='Filter vehicle model (default: "vehicle.*")'
    )
    argparser.add_argument('--generationv',
        metavar='G',
        default='All',
        help='restrict to certain vehicle generation (values: "1","2","All" - default: "All")'
    )
    argparser.add_argument('--car-lights-on',
        action='store_true',
        default=False,
        help='Enable automatic car light management'
    )
    argparser.add_argument('-m', '--map',
        default='Town10HD_Opt',
        help='load a new map, use --list to see available maps'
    )
    argparser.add_argument('-n', '--number-of-vehicles',
        metavar='N',
        default=5,
        type=int,
        help='Number of vehicles (default: 30)'
    )
    argparser.add_argument('--safe',
        action='store_true',
        default=True,
        help='Avoid spawning vehicles prone to accidents'
    )
    argparser.add_argument('--asynch',
        action='store_true',
        default=False,
        help='Activate asynchronous mode execution'
    )
    argparser.add_argument('--hybrid',
        action='store_true',
        help='Activate hybrid mode for Traffic Manager'
    )
    argparser.add_argument('--no-rendering',
        action='store_true',
        default=False,
        help='Activate no rendering mode'
    )
    argparser.add_argument('-s', '--seed',
        metavar='S',
        type=int,
        help='Set random device seed and deterministic mode for Traffic Manager'
    )
    argparser.add_argument('-dir', '--directory',
        # default='C:/Users/ngoak/Projects/carla/PythonAPI/visionor',
        default='./data',
        help="directory for saving images"
    )
    args = argparser.parse_args()
    random.seed(args.seed if args.seed is not None else int(time.time()))
    logging.basicConfig(format="%(levelname)s: %(message)s", level=logging.INFO)
    client = carla.Client(args.host, args.port)
    client.set_timeout(10.0)
    synchronous_master = False
    # Load the map    
    print('load map %r.' % args.map)
    world = client.load_world(args.map)
    try:
        world = client.get_world()
        traffic_manager = client.get_trafficmanager(args.tm_port)
        traffic_manager.set_global_distance_to_leading_vehicle(2.5)
        if args.respawn:
            traffic_manager.set_respawn_dormant_vehicles(True)
        if args.hybrid:
            traffic_manager.set_hybrid_physics_mode(True)
            traffic_manager.set_hybrid_physics_radius(70.0)
        if args.seed is not None:
            traffic_manager.set_random_device_seed(args.seed)
        settings = world.get_settings()
        if not args.asynch:
            traffic_manager.set_synchronous_mode(True)
            if not settings.synchronous_mode:
                synchronous_master = True
                settings.synchronous_mode = True
                settings.fixed_delta_seconds = 0.01
            else:
                synchronous_master = False
        else:
            print("You are currently in asynchronous mode. If this is a traffic simulation, \
            you could experience some issues. If it's not working correctly, switch to synchronous \
            mode by using traffic_manager.set_synchronous_mode(True)")
        settings.no_rendering_mode = args.no_rendering
        world.apply_settings(settings)
        ################################
        vehicles_list = spawn_vehicles(world, client, args, synchronous_master)
        camera_list = spawn_cameras(world, client, args, synchronous_master)
        # camera_list = []
        count = 0
        while count < 1000:
            count += 1
            if not args.asynch and synchronous_master:
                world.tick()
            else:
                world.wait_for_tick()
    except Exception as error:
        print("error")
        print(error)
    finally:
        if not args.asynch and synchronous_master:
            settings = world.get_settings()
            settings.synchronous_mode = False
            settings.no_rendering_mode = False
            settings.fixed_delta_seconds = None
            world.apply_settings(settings)
        time.sleep(2)
        print("\ndestroying %d vehicles" % len(vehicles_list))
        client.apply_batch([carla.command.DestroyActor(x) for x in vehicles_list])
        print("\ndestroying %d camera" % len(camera_list))
        for camera in camera_list:
            camera.destroy()
        time.sleep(2)  # finishing saving the last images

if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        pass
    finally:
        print("\ndone.")

[STARKEY13]

The default timeout of 10 seconds might not be sufficient when using multiple cameras. Increase the timeout to give the simulator more time to process the requests

[LeonardMendicantBias]

Thank you for the quick response. Unfortunately, the error persists. I modified the following. client.set_timeout(50.0) The simulations simply wait 50 seconds before showing the same error.

One more "anomaly", I switch the simulation to async mode and everything works. However, this would not do since I need the synchronous mode.

[PatrickPromitzer]

Hi, the "time-out" error happens if Carla didn't answered. It could be the connection, Carla crashed, Carla took too long, or everything else that could prevent a answer from the server in the timeout limit.

You can try to add some debug messages to see where the script got an error. Did the Server still run after the time-out error?

btw. the code is hard to read because of the formatting. You can use

   ```python
    code
    ` ` `

to keep the format. (without space)

[LeonardMendicantBias]

@PatrickPromitzer Thank you for the coding quote tips. I did not know about triple ``. The github's editing tool only shows single. I edited the code for future reading.

The UE 4 editor is running while the script is waiting for a response! After the time-out message, I can still move around in the UE 4 editor (using WASD). However, If I start the script again, the script returns the time-out message without the initial images. I think that using 4 cameras crashes the Carla server and prevents later connections. If I close the UE 4 editor and reopen it, the script still cannot find the Carla server. Does that mean the Carla server still running and the reopened UE 4 editor started another server? Since the UE 4 editor is still "functional", I really don't know where to debug. Do you have any suggestion where should I watch for a possible bug?

p/s: I start the UE 4 editor in the folder "./Unreal/CarlaUE4/CarlaUE4.uproject" after I finished build Carla from source.

[PatrickPromitzer]

Normally, if the server crashes the unreal editor crashes, too. (maybe only on Ubuntu) You could search if an old instance is running and close everything Carla related. It is not a solution, but maybe something in the background is creating a problem.

You could look if your firewall is blocking something.

You could try making a package and test it with that package.

One thing that helps for me is adding "world.tick()" in between adding sensors or listeners.

You could try using one blueprint instance for each camera (just to be sure, even if should not make a difference)

You can watch your CPU/GPU usage, too.

For

camera_1.listen(lambda image: image.save_to_disk(f'{args.directory}/camera-1/{image.frame:06d}.png'))

you could call an custom function and log the function call.

with open("log_file.txt", "a") as f:    
    f.write("Function call: {0}".format(camera_name))

Maybe I come up with more ideas at a later time.

[LeonardMendicantBias]

@PatrickPromitzer Thank you so much for your suggestions. My firewall has been turned off. I do have a defined function for the callback.

I don't know what I have changed, but somehow the script works. I mark this as solved but I really don't know what was the problem.

[LeonardMendicantBias]

I found the problem. It is the sensor_tick. When I set the value too low (i.e., 0.02 for 50 fps), the simulator time-out while it works well with sensor_tick=1. I guess I need a better computer.

[PatrickPromitzer]

The normal tickrate is 0.1, and at some point, the documentation said you should not go below 0.05

Try which rate works for you before buying new hardware.

[LeonardMendicantBias]

Yes, the document states that the simulation should not be less than 10 FPS (0.01) or the physic would get weird. But sensor_tick is the update rate of the camera, not the simulator itself.

Also, I am able to utilize sensor_tick=0.02 by adding the image into a global queue in the callback function. Then, after the world.tick(), I save the queued images. This makes the simulations not real-time anymore, but that does not matter since I only need the synthetic data.