Get further insights for the Simulation notebook

[FrancescoMandru]

In the Simulation notebook you initially say that it's crucial to simulate other agents to remove false positive interventions which occour when agents are log-replayed what do you mean by log-replay?. To do so you provide a pre-trained network for agents and one for the SDV. Then you load only the agent dataset and you run a closed-loop evaluation and then you unroll it (It will be fantastic to get more details about this part).

Then you provide a quantitative evaluation for the agents looking at the ADE (how can you do that if you are looking only at the EgoDataset?, Does it means that you are able to completely access agent information also from the Ego?)

Then you provide a qualitative evaluation.

After this part you measure the ego performance with simulated agents. To do so you look at very interesting metrics like collsion etc. Can you be more specific about the process of looking at these metrics and how precisely you simulate the scene to compute them? This is something that can be used during the training to penalize a trajectory that collide with other agents?

At the end there is a table for the comparison but it returns always zero for every field

[lucabergamini]

what do you mean by log-replay?

This is when you use agents in the dataset. Those agents have been recorded in the real world but they don't react to any changes the planner take

Then you load only the agent dataset and you run a closed-loop evaluation and then you unroll it

We actually use an EgoDataset only. then:

• we rasterise ego for the ego model (planner);
• we rasterise all agents using custom calls for the agent model (simulator) see here

how can you do that if you are looking only at the EgoDataset?, Does it means that you are able to completely access agent information also from the Ego?

We can call the get_frame function for the EgoDataset for agents too. The AgentDataset inherits from the EgoDataset in the end and call the parent's functions.

Can you be more specific about the process of looking at these metrics and how precisely you simulate the scene to compute them?

Can you be more specific about what you want to know here?

This is something that can be used during the training to penalize a trajectory that collide with other agents?

you would need to mix simulation and evaluation, it's really tricky.

At the end there is a table for the comparison but it returns always zero for every field

You will need to simulate a lot of scenes for a lot of frames to see differences between the two modes. The planner is driving ego also for the agents log replay

[lucabergamini]

can I close this @FrancescoMandru ?

[FrancescoMandru]

I close it for the moment and I will reopen it in the future

[FrancescoMandru]

Hello @lucabergamini. I adapted the closed loop code for my Network, at the end the predicted position and yaw is returned as in the standard implementation. I studied better your notebook example and I have some doubts.

In this setting, a chasing car will not slow down if our policy choses a different speed profile for the SDV, resulting in a rear collision that wouldn't obviously occur in the real world.

For this reason, closed-loop evaluation is only accurate for the first few seconds of forward simulation. This can be mitigated when enough data exists for the task.

Is this the case?

num_scenes_to_unroll = 10 num_simulation_steps = 50 NOTE: these properties have a significant impact on the execution time. We suggest you to increase them only if your setup includes a GPU

Could you be more specific about the tuning of these parameters? I use num_simulation_steps = 40 which is the number of steps used for training my model. Regarding num_scenes_to_unroll which is a statistical significal value?

[FrancescoMandru]

Moreover, I used your example notebook setting num_scenes_to_unroll=5000 and the notebook crashes. This is probably due to the fact that you are passing to the model a batch with a dimension equal to num_scenes_to_unroll (maybe) which is prohibitive.

[lucabergamini]

This is expected. If you need to unroll a lot of scene wrap the current code in a for loop to limit the batch size. Otherwise, a high number of scenes will be loaded in memory prior to start unrolling