Epic: More flexible architecture for networking communication

[nsubiron]

Update: a work-in-progress version of the new design can be found in the carla-0.9 branch.

Introduction

There are few key features that we want to add to CARLA

• Multiple clients connecting to the same simulator
• Ability to open any city on demand and possibly generate cities at runtime based on some definition format #65
• Ability to control all the agents in the scene #141 #184 #428
• Ability to spawn other agents on demand #14
• Ability to control agent by different methods (e.g. gas+steering+brake vs teleport to location) #270 #280
• Ability to place sensors anywhere or attached to any agent #59
• Better filtering of the info received about agents in the scene (e.g. agents within FOV) #63

Here by agents we refer to all the vehicles, pedestrians, traffic lights/signs, and other things that could be added in the future that are present in the scene; basically anything that can be controlled and affects up to some extent the driving.

Implementing these features with the current design of the networking communication between the client and the simulator would be near impossible. As it is right now the network protocol is very strict and not at all fail proof, it is assumed everywhere that there is a single client making request at a certain timely order. Given this, it seems like the perfect time for designing an scalable and fail proof architecture that can serve us in the long run.

So I'm proposing the following requirements that the new architecture should comply with

Main design requirements

First of all, there will be two different ways a client can communicate with the server, 1) general requests and 2) data streaming (optimized for speed)

At least in the first version, the streaming will be done via TCP and not UDP as shown in the diagram.

1) Request-response model for controlling the simulation

• A client connects to the server's main port and sends a request
• Every request from a client has a response from the server
• Any request can be sent at any time by any client
• The server can always respond with a failure (not possible, not implemented, not enough permissions, etc.)
• Connection might be broken at any time and the simulation should carry on
• At any time any client (with enough permissions) can request a new episode giving a map definition or the name of a predefined map (level 1)
• At any time any client (with enough permissions) can change traffic related objects without changing the episode (level 2) (let's see how feasible is this with UE4)
• At any time any client (with enough permissions) can spawn a new agent anywhere (level 3)
• At any time any client (with enough permissions) can spawn a new sensor anywhere, or attached to any agent (level 3)
• Any client can control any number of agents
• Each agent has a single owner client, only its owner can send control messages
• Everything should be recorded so we can do a replay

2) Data streaming for sending the sensor data

• First by method 1 a client requests a sensor, the server responds with directions to connect to the data stream
• The client subscribes to the sensor and get a stream of data every frame (or every few frames)

Where the level 1, 2, and 3 refer to the different definitions of an scenario

• Level 1: Static layout and objects
• Level 2: Traffic rules and related objects (signs, traffic lights, etc)
• Level 3: Dynamic agents behaviour

Use example

• Client A connects to server, requests status, status is "waiting for a map definition"
• Client A requests a new episode by giving a map definition, server generates and loads the map, responds OK (at this point by the way UE4 works the simulation starts)
• Client A requests the creation of a set of traffic rules/agents (traffic lights, speed limit signs, etc), the server adds them to the scene and responds OK
• Client A requests 50 pedestrians by giving their spawn transforms, models, etc, the server adds them to the scene (but they won't move yet) and responds OK
• Client B connects and requests the agents of type pedestrian that are not controlled by any client, the server responds with a list of the 50 pedestrians
• Client B requests controlling those 50 pedestrians, server says OK
• From this point on, client B sends the control of the pedestrians each frame
• In parallel to this, client C requests a new vehicle at a given location (and in the same requests it asks to control it), the location is occupied, server responds with error and possibly a nearby available location
• Client C requests again a vehicle at the new location, server says OK
• Client C sends a control message to that vehicle each frame
• Client D connects, requests control over traffic lights and starts changing their state
• etc.

Open questions

• How are we going to deal with synchronous mode?
• How does a client get the info about agents? make a "sensor" that monitors certain agents?
• There are going to be races between clients to e.g. get control of agents, → first-in wins?
• Should we keep using protobuf? other options: rpclib, own serialization, ...
• Should an "autopilot" be available in the server side?
• Can we implement it inside Unreal or as external lib as it is now?

Changes required in Unreal-side code

• There is no "player vehicle" anymore
  • Instead an spectator player that can move around and follow agents
  • Every agent is an NPC → Important changes in the Controllers
• It has to be possible to change the map from the client side
  • Simulator starts on a transition map, loads the maps on request
• There has to be an easy way of retrieving and filtering all the agents in the scene
• The simulator should be able to generate a map from a map definition (long term)

[felipecode]

How would be an example interaction from the client side with the new architecture ? This will help us on the definition of #428, client side agents. For instance right now we have this interaction:

with make_carla_client(args.host, args.port) as client:
    settings = CarlaSettings()
    scene = client.load_settings(settings)
    client.start_episode(player_start)
    for i in range(number_of_iteration):
        measurements, sensor_data = client.read_data()
        # Do something with measurements
        client.send_control()

How is this going to change ?

[nsubiron]

@felipecode It is still early as to say something definitive, but what I have in mind is something like this

client = CarlaClient("localhost", 2000)

client.start_episode(CarlaSettings())

vehicle = client.spawn_vehicle("mustang", Transform(x=100.0, y=100.0))
camera = client.spawn_camera(attach_to=vehicle.id, Transform(z=50))

for i in range(number_of_iteration):
    vehicle.apply_control(throttle=0.5, steer=0.0)
    image = camera.read_data()

[felipecode]

Amazing @nsubiron . I like how we are going to remove the concept of the client receiving and sending control commands. Now, this is a task of vehicles and sensors. Take into account that many of these objects can exist into world. I will consider this on #428 .

[analog-cbarber]

1) Request-response model for controlling the simulation ...

• Everything should be recorded so we can do a replay

I am interested in being able to replay a session with the exact same actor positions but with different lighting, weather, or sensor configuration. Is that the kind of activity you intend to support? Will this work include an API for actually running a session from recorded actor poses, or is that for future work?

[nsubiron]

Yes, that's the kind of things we want to support in the future. For the moment (this issue), I'm just implementing the infrastructure to allow finer control over every actor in the scene.

For the record, you can already reply an episode with the same position of other vehicles and pedestrians but different weather or sensors. Just that the behaviour of NPCs is controlled by a random seed.