woven-planet / l5kit

L5Kit - https://woven.toyota
https://woven-planet.github.io/l5kit
858 stars 277 forks source link

Performance of trained Urban Driver model poorer than pretrained model #386

Open jeffreywu13579 opened 2 years ago

jeffreywu13579 commented 2 years ago

Hi, I was training the open loop planner with ego history model using the notebook provided in the urban driver folder for up to 800k iterations with the default config settings. In the first ~50k iterations, my training loss converged to around 0.048 and stayed roughly around that, before going up to around 0.12 after 600k iterations. Using the closed loop test notebook also in the urban driver folder, I evaluated my model at 500k iterations (when training loss was around 0.05) and at 800k iterations (when training loss was 0.12). For both models, the displacement error and collisions (particularly rear) was significantly higher (up to 5+ times more collisions) compared to the pretrained model (OL_HS.pt) provided in the training notebook. Am I missing anything configuration/hyperparameter changes needed for training?