search

motional / nuplan-devkit

The devkit of the nuPlan dataset.
https://www.nuplan.org
Other
674 stars 129 forks source link

Trajectory in nuboard deviates from model output #202

Closed mh0797 closed 1 year ago

mh0797 commented 1 year ago

Describe the bug

The trajectory displayed in the nuboard differs from the trajectory that is generated by the trained model when using the scenario as input and the one output by the ML planner.

Setup

I ran an open loop simulation for my planner. The trajectory displayed in the nuboard seemed weird to me (oscillates very heavily even though the model appears to be very robust on the validation set) . So I tried to recreate the trajectory in two ways:

  1. I loaded the scenario and generated the features from it. After that I infered the model.
  2. I instantiated the Simulation object myself and created the planner input from the scenario. Finally I evaluated the MLPlanner which was configured to use the trained model. The features generated in both cases are equal (so there is no error in generating features from simulation compared to generating them from scenario). The resulting trajectory differs significantly from the one seen in nuboard.

Question

Is there any postprocessing done to the trajectory that is output by the model which could cause this significant difference? If not, what could potentially be the reasons for this deviation?

Note: this issue affects not only a single scenario but almost every single one in the simulation.

mh0797 commented 1 year ago

I just found out that this was a nasty bug within my model. I am closing the issue and apologize for having bothered you.

mh0797 commented 1 year ago

Unfortunately the issue is occurring again, this time it seems to not be caused by the model. Can you tell me what exactly the nuboard shows ? Is it the model output that is shown or is the model output post-processed in some way before being visualized?

patk-motional commented 1 year ago

Hi @mh0797,

Sorry for the late reply. What mode are you simulating? If you are using the closed-loop simulation it will utilize and motion model and kinematic controller to follow your trajectory. This could explain he deviations.

mh0797 commented 1 year ago

Hi @patk-motional,

I was able to find the reason for the deviation. In Training all trajectories (ground truth / model output) are given as a sequence of rear axle poses. What was confusing me, is that the nuboard does not visualize the model output directly, but instead the Cog-trajectory.

For straight driving, especially at medium and large velocities, these are very similar. However, at low speeds there can be significant deviations between the cog-trajectory and the rear axle trajectory.

With that said, i will close this issue. Thank you for your support!