first stab at replay

[bradyz]

this branch is not ready to merge, just wanted to start some discussion on two questions -

thoughts on how to configure sensors? I'm thinking of creating yaml files that have the configurations of all the sensors (sensor type, pose, etc)...

Also, how should we save certain data types? Right now there are only RGB cameras, and the only information saved is the frame tick that the image was captured from - we should also save the pose at every time step.

[philkr]

I'd move most of this into recording. This way we can keep the top hierarchy slimmer.

[philkr]

A few comments on the structure:

• I'd use config files, similar to the Scenario
  • Sensors settings are fully specified in the config YAML file (e.g. FPS aka frame skip, resolution, FOV, relative angle, ...)
  • Sensors can be attached to vehicles, other agents, or live freely in the world
  • We should allow for variables in the config file:
    • The main vehicle should have a certain identifier in the config file (i.e. some variable $MAIN_VEHICLE)
    • This identifier should be replaceable with a command-line argument (e.g. -s 'MAIN_VEHICLE=123') for the exact vehicle in question.
    • The same should be used for other things like location of the main camera if static, ...
    • Consider defining default values for variables somehow (if at all possible in YAML)
  • Each sensor can produce one or more named outputs (i.e. image stream, odometry stream, ...)
    • When recording data, we specify an output base (e.g. /tmp/my_first_recording_), but record into multiple files: one per named output (e.g. /tmp/my_first_recording_dashboard.mp4 or /tmp/my_first_recording_dashboard_0001.jpg), plus one that contains the config YAML with all variables replaced (e.g. /tmp/my_first_recording_config.yaml).
    • Allow for a directory base: /tmp/my_first_recording/ -> /tmp/my_first_recording/config.yaml (remember to Path(config_output_file).parent.mkdir(parent=True))

No need to go over board with all options for now as long as we have an easy to extend framework. We can always add more options in later PRs as needed.

[bradyz]

here's some sample usage with using sensor config files

I have a hack for variables that i can whip up tomorrow.

and addressing the named outputs - this /tmp/my_first_recording_* seems like it could get messy - why not just have the structure like

/tmp/my_first_recording
    rgb
    rgb_left
    rgb_right

i.e. give each sensor access to the /tmp/my_first_recording directory, and do whatever they want inside here?

[bradyz]

did a big overhaul and ready for a new review - i think the structure is pretty stable now,
but certain implementations (the parsing json) could use a little work

[philkr]

@bradyz I rewrote most of the configuration system and tried to remove as much of the duplicate code as possible. I only tested the recording for now, and didn't test replay yet. There are a few things that remain to be done:

• [ ] Add sensors to replay when needed
• [ ] (optional) Record sensors without replay (maybe through the record.py or a separate script)
• [ ] Test and debug this
• [ ] Write some sort of documentation on how to use the configs and different context managers
  • Right now a config file can consist of multiple yaml files each containing a config for either sensors, scenario, rendering
  • You can load as many files as you want, but can only define a single scenario and rendering option
  • To create a new config, just create a class with type decorators and register it. Configuration will take care of the rest.

[bradyz]

i like the intense revamp of the config files - i've made a couple small changes to get sensors into replay

some issues remain:

• need some sort of synchronization between the sensors and the world (right now the replay can terminate without the sensors finishing up and only writing some of the files to disk)
• we should also have an optional output stream which outputs the parsed sensor data some native python object (we can use this in the future if we want to rollout an agent)
• config overrides

right now i run this via

python3 -m carla_utils.replay town03.log tmp/ \
    --config config/sensors/dashboard_multi.yaml \
    --config_override "[setattr(s, 'target_actor', 2060) for s in sensor]" "[setattr(s, 'output_format', 'png') for s in sensor]"

[philkr]

What is the sync issue? I don't really get it.

output stream: I think we shouldn't overthink this. How about we just return a named list of sensors from the sensors function, allow output_format to be None, and cache the last output in each sensor? After all any code that accesses the sensors will have access to the context manager anyway.

I think config overrides are close. The setattr style is ugly though (I didn't even thing of that). I was thinking we can use ellipses and make sensors a special list to hangle this. Then we can use sensor[...].target_actor = 2060

[bradyz]

the callbacks are handled in a separate python thread (i think)

so even as the world ticks on, the sensors callbacks can be lagging behind
i run into the issue where the world has ticked all the way to the end of the recording, terminating the python program, but the sensors are not yet "caught up" and have not finished writing all the sensor data to disk

[philkr]

The callback threads are from carla itself? We should look into that.

[bradyz]

I don't know too much about how it's implemented - I took a look a couple days ago but didn't get much out of it.

At the end of the day for true sync mode we ll just need a barrier or something that waits for the sensors to get caught up with the world ( I implemented a crappy version of this right before the config rewrite)

[bradyz]

added the sensor warning + try/finally

let's take a final look soon then squash + merge