Record/Playback and synchrony

[raymanfx]

Hello,

I have a question regarding the record/playback feature of CARLA with regards to synchrony. My goal is the following: I have an ego vehicle which is controlled by a behavior agent, driving around in a certain map. I enable the synchronous mode [1] for this and use CARLA's server side recording feature to be able to reproduce the scene.

When playing back the scene, it is not clear to me whether I can "force" synchronous mode as well to control the playback time step etc. manually. Currently, I'm using wait_for_tick() which means the playback is async, although the recording itself was done in sync mode.

I require sync mode for the recording so my behavior agent does not lose information when the server runs faster than the client. In playback mode, I need to add three (or more) RGB cameras, which obviously destroys the FPS. Thus, I would like playback to be synchronous as well and save the images of each respective camera to disk - then proceed with the next playback step in a loop.

TL;DR can I enforce synchronous mode for the server-side playback feature via the client?

[1] https://carla.readthedocs.io/en/latest/adv_synchrony_timestep/

[xinshuoweng]

any update on this?

[raymanfx]

AFAIK this is not possible in CARLA right now. Unfortunately that makes the recording feature much less useful :(

[xinshuoweng]

I see. Thank you! BTW, how do you even use wait_for_tick() during replay? From the examples, I can see that we can use client.replay_file() function to replay the recording, but how it is possible to add a loop inside and tick at every timestep of the replay?

[PyTorr]

Why do you need replay for this? You can add all the cameras in one recording.

[xinshuoweng]

There could be many usages of this. One good reason is for reproducibility, i.e., I would like to be able to record the exact same data (camera, lidar, etc) using the replaying function

[corkyw10]

@DSantosO are you able to help with this please?

[raymanfx]

Why do you need replay for this? You can add all the cameras in one recording.

Running the simulation with 3 (or more) camera sensors attached is very computationally expensive. I would like to be able to record the scenery with an autonomous agent driving the vehicle. When the route navigation is completed, I review the recording to make sure the car did not crash / drive into a wall / run over pedestrians, etc. If the recording is good, I attach the sensors reenact the scene to get the additional data.

[PyTorr]

Good reason. How do you show the replay?

On Fri, Mar 5, 2021 at 4:16 PM Christopher N. Hesse < notifications@github.com> wrote:

Why do you need replay for this? You can add all the cameras in one recording.

Running the simulation with 3 (or more) camera sensors attached is very computationally expensive. I would like to be able to record the scenery with an autonomous agent driving the vehicle. When the route navigation is completed, I review the recording to make sure the car did not crash / drive into a wall / run over pedestrians, etc. If the recording is good, I attach the sensors reenact the scene to get the additional data.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/carla-simulator/carla/issues/3624#issuecomment-791445713, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACDB7A22XTWDUQAGP2VEVZDTCDRSNANCNFSM4UAZ6OSQ .

[PyTorr]

I did it with synchronous mode. It's slow, but it works.

[Axel1092]

Hi all, sorry for the late response. The recorder and the synchronous mode feature can be mixed in any way you want (although for precise measurements we suggest to always use synchronous mode). You can run the recorder for any amount of time in asynchronous or synchronous mode and then replay it with synchronous mode. Before starting the replayer, set the server to synchronous mode, then run the replay function and run every tick with the world.tick() functions instead of wait_for_tick().

During the replay you can even create new sensors to obtain new information if you need it.

[xinshuoweng]

Hi Axel, thanks for the information. I am wondering is there any code piece for the function you mentioned? In the documentation, there are only a few simple functions introduced here about how to replay. However, there is no instruction code about how to create sensors during replay

[Axel1092]

There is no specific code because it works the same as during normal simulation, you should do something like this:

1. Enable synch mode with desired settings and delta time
2. run client.replay_file("recording01.log", start, duration, camera)
3. world.tick() This starts the replaying typically spawning the initial actors
4. Spawn new actor/sensor/whatever world.spawn_actor(blueprint)
5. Simulation loop:

   whicle(True):
   world.tick()
   #your code

[PyTorr]

Very cool I will try it.

For now I'm only able to save the logs on the predefined location in the Server. I can't change the path, which is a problem for me because I'm running the server on another computer that I can't access.

[raymanfx]

This sounds very promising, I'll try it out again today.

[raymanfx]

Okay, so it's .. somehow working but the overall setup feels a bit flaky. Here's what I did:

1. Start the server
2. Spawn a vehicle, use an agent to drive along a route, record the scene using the Carla recorder

Now for the playback part:

1. Start the server
2. Start a "dummy" client that plays back the recording and spawns a camera

The code for this "dummy" client boils down to this:

    args.sync = 30
    ...

    client = None
    world = None
    carla_world = None

    try:
        client = carla.Client(args.host, args.port)
        client.set_timeout(4.0)

        carla_world = client.get_world()

        if args.sync:
            fps = args.sync
            step = 1.0 / fps
            print("Enabling sync mode with fixed time step: {}s".format(step))
            settings = carla_world.get_settings()
            settings.synchronous_mode = True
            settings.fixed_delta_seconds = step
            carla_world.apply_settings(settings)
            # initial tick() before initializing the agent
            carla_world.tick()

        client.replay_file("aagent_control.rec", 0.0, 0.0, 0)

        # Get all current actors and choose a parent.
        vehicle = None
        actors = carla_world.get_actors()
        for actor in actors:
            if actor.type_id.startswith('vehicle'):
                vehicle = actor
                break

        if not vehicle:
            print('[ERROR] Failed to find vehicle actor')
            raise ValueError

        ...
        // spawn a camera and attach it to the actor
        ...

        while True:
            if controller.parse_events():
                return

            if args.sync:
                # Signal the server to calculate the next frame
                carla_world.tick(10.0)
            else:
                # As soon as the server is ready continue!
                if not world.world.wait_for_tick(10.0):
                    continue

            // render camera using pygame

Does this look fine to you? One issue I'm facing is that sometimes the actor is found, sometimes not and the ValueErrror is thrown. I guess client.replay_file is not blocking so the scenery is not set up completely at that point. Inserting a sleep(..) could probably work around that, but perhaps there's a better way?

[raymanfx]

I believe the "empty actor list" issue is linked to this: https://github.com/carla-simulator/carla/issues/3945, where I documented a workaround.

[Axel1092]

@raymanfx, your issue of the actor not being found when you do the actors = carla_world.get_actors() is probably due to the fact that the server has not yet spawned the vehicles. After client.replay_file("aagent_control.rec", 0.0, 0.0, 0) the server has not yet replayed anything, you need to call carla_world.tick(10.0) so that it can start processing the recorded sequence which among other things spawns the vehicles. Other than this the code is what I would do here, just add a carla_world.tick(10.0) just after replay_file.

[egeonat]

There is no specific code because it works the same as during normal simulation, you should do something like this:

1. Enable synch mode with desired settings and delta time

2. run `client.replay_file("recording01.log", start, duration, camera)`

3. `world.tick()` This starts the replaying typically spawning the initial actors

4. Spawn new actor/sensor/whatever `world.spawn_actor(blueprint)`

5. Simulation loop:

whicle(True):
    world.tick()
    #your code

@Axel1092, I followed these instructions and wrote my code that looks very similar to what raymanfx has shared. The issue is that the replay_file function changes the simulation settings to be asynchronous again. So that when I do:

print(world.get_settings().synchronous_mode)
client.replay_file(record_path, 0, 0, 0)
print(world.get_settings().synchronous_mode)

My output is first True, and then False. The simulation then starts moving without waiting for my world.tick() calls. Is there a way to start replaying in synchronous mode from the start?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.