Why is only the 0-th iteration of each scenario used for training?

[mh0797]

I noticed the feature and target generators to use only iteration 0 of each scenario. For instance in the ego_trajectory_target_builder extracts the trajectory by

scenario.get_ego_future_trajectory(
    iteration=0,
    num_samples=self._num_future_poses,
    time_horizon=self._time_horizon
)

Is there a reason that the other iterations of the scenario are unused? And is it possible to generate features for subsequent frames of the same scenario, for instance in order to check for temporal consistency of a planner?

[patk-motional]

Hi @mh0797,

Each scenario for training - which I will refer to it as a sample for this discussion - is anchored by a starting token in a log. You are able to access past and future information from that anchor token.

When generating the data going into the data loader, samples are created by sampling anchor lidar pcs from the DB files. We have filters that specify how frequently to sample the data (scenario_filter.limit_total_scenarios=0.1 equals to sampling 1 sample from every 10 samples) or how far apart each sample should be (scenario_filter.timestamp_threshold_s=10s will ensure that there are at least 10s between each sample).

And is it possible to generate features for subsequent frames of the same scenario, for instance in order to check for temporal consistency of a planner?

The short answer, technically it is possible, but not supported. If you do not set scenario_filter.timestamp_threshold_s then chances are you are already consuming samples that are temporally near to each other. However, we have no supported way to directly inspect two samples and subsequent samples during feature computation and training.

You can try to create visualization callbacks and log them in tensorboard. Hope that helps