search

MCZhi / DIPP

[TNNLS] Differentiable Integrated Prediction and Planning Framework for Urban Autonomous Driving
https://mczhi.github.io/DIPP/
197 stars 40 forks source link

Queries about the final validatioin loss #5

Closed snwmbgsct closed 1 year ago

snwmbgsct commented 1 year ago

Interesting work! May I know your final validation loss? My result is far worse than your demo, I am confused. Thanks~

Here is my validation loss:

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

epoch | loss | lr | val-loss | train-plannerADE | train-plannerFDE | train-predictorADE | train-predictorFDE | val-plannerADE | val-plannerFDE | val-predictorADE | val-predictorFDE -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- 1 | 5.30640924 | 0.0002 | 3.35200429 | 1.98499836 | 5.19161138 | 2.27674091 | 4.44882256 | 1.48450993 | 3.90581918 | 1.21964474 | 2.59888194 2 | 3.39305077 | 0.0002 | 3.15594395 | 1.44424248 | 3.81059549 | 1.23855729 | 2.65948786 | 1.54844272 | 4.137977 | 1.08765502 | 2.352332292 3 | 3.08943794 | 0.0002 | 3.01338992 | 1.27482148 | 3.36427458 | 1.08588636 | 2.357684 | 1.29808065 | 3.51700335 | 1.08436095 | 2.267689138 4 | 2.93844681 | 0.0002 | 2.82100026 | 1.2168865 | 3.20476923 | 0.99612185 | 2.17641783 | 1.17624403 | 3.12159375 | 0.91968476 | 1.998957871 5 | 2.6849731 | 0.0001 | 2.60172415 | 1.08432435 | 2.82212017 | 0.86019267 | 1.94155749 | 1.06294514 | 2.78237748 | 0.7684531 | 1.781172586 6 | 2.98904202 | 0.0001 | 4.001105 | 1.86916319 | 4.82792364 | 0.85137051 | 1.92989603 | 1.36248824 | 3.64402256 | 0.7933633 | 1.790358165 7 | 2.7280053 | 0.0001 | 3.87203419 | 1.28932238 | 3.34680784 | 0.82810459 | 1.87784089 | 1.3214879 | 3.47108813 | 0.75594594 | 1.782241155 8 | 2.70244375 | 0.0001 | 3.7138443 | 1.27081805 | 3.29782817 | 0.81973241 | 1.85990532 | 1.23529392 | 3.26991886 | 0.76108522 | 1.735746209 9 | 2.57701064 | 5.00E-05 | 3.51180987 | 1.18620428 | 3.07102187 | 0.76134746 | 1.75147484 | 1.16045334 | 3.0205913 | 0.69535275 | 1.64322956 10 | 2.55585605 | 5.00E-05 | 3.53438971 | 1.18744692 | 3.02984186 | 0.75051616 | 1.72726152 | 1.19285512 | 3.10802594 | 0.6888317 | 1.617973151 11 | 2.53238232 | 5.00E-05 | 3.63254311 | 1.17100767 | 3.00655693 | 0.74324979 | 1.70910997 | 1.21436171 | 3.2409484 | 0.69732279 | 1.628038934 12 | 2.5160189 | 5.00E-05 | 3.56046366 | 1.16027349 | 2.9606347 | 0.73646933 | 1.69483182 | 1.19161167 | 3.16818277 | 0.67683438 | 1.619803391 13 | 2.46179123 | 2.50E-05 | 3.38446206 | 1.12082525 | 2.85831879 | 0.71113592 | 1.64634189 | 1.12828799 | 2.90106323 | 0.65835266 | 1.566396024 14 | 2.4487983 | 2.50E-05 | 3.36255161 | 1.12622047 | 2.85643034 | 0.70456014 | 1.6292965 | 1.11457964 | 2.89173817 | 0.65227702 | 1.560581859 15 | 2.43642483 | 2.50E-05 | 3.34954591 | 1.11324575 | 2.82192018 | 0.70032796 | 1.61985126 | 1.13522416 | 2.90769757 | 0.6440806 | 1.548792343 16 | 2.42463908 | 2.50E-05 | 3.35748811 | 1.10034413 | 2.79777732 | 0.69747301 | 1.61327508 | 1.13133556 | 2.89095114 | 0.64437464 | 1.538969017 17 | 2.39755128 | 1.25E-05 | 3.31509354 | 1.08073514 | 2.73777261 | 0.68414007 | 1.58632264 | 1.11879283 | 2.84425474 | 0.63509695 | 1.522461395 18 | 2.38913355 | 1.25E-05 | 3.29188321 | 1.08298306 | 2.71589887 | 0.68159302 | 1.57892312 | 1.09898604 | 2.77768362 | 0.64100605 | 1.530593221 19 | 2.38619561 | 1.25E-05 | 3.27823806 | 1.08456751 | 2.71279631 | 0.67825057 | 1.5717595 | 1.09826435 | 2.80898136 | 0.63461334 | 1.517605718 20 | 2.392062 | 1.25E-05 | 3.2462033 | 1.07611374 | 2.7040438 | 0.68312093 | 1.58415749 | 1.0818814 | 2.75898347 | 0.62684758 | 1.503441528 21 | 2.37400019 | 6.25E-06 | 3.21383737 | 1.05727336 | 2.6592687 | 0.67698563 | 1.57042057 | 1.06992927 | 2.70909504 | 0.62652129 | 1.498268957

MCZhi commented 1 year ago

Thank you for your interest in our work. Your results look good to me and they are similar to my work. What demo you are referring to?

MCZhi commented 1 year ago

How many data points did you use to train the model? Also, I would like to suggest starting the planner when the learning rate decreases to 5e-5, which may lead to better results.

snwmbgsct commented 1 year ago

Thank you for your interest in our work. Your results look good to me and they are similar to my work. What demo you are referring to?

In the Project Website

How many data points did you use to train the model? Also, I would like to suggest starting the planner when the learning rate decreases to 5e-5, which may lead to better results.

127,548 for training and 4,615 for validation. We will give it a shot again.

Attached are videos reproduced by us. Thanks for any comment.

https://user-images.githubusercontent.com/72658280/186641483-b0d27f04-89da-4977-9e60-91c58c12bedd.mp4

https://user-images.githubusercontent.com/72658280/186641497-f968d7df-07d1-4d95-a55b-ce12c14c847d.mp4

https://user-images.githubusercontent.com/72658280/186641510-29fd04fd-4468-4359-8622-8f0a4c70bc48.mp4

https://user-images.githubusercontent.com/72658280/186641519-032fcdca-2b35-4911-a9eb-080576168dab.mp4

MCZhi commented 1 year ago

Yeah, that is bad. I guess there might be something wrong with the training process. My final validation loss for the planner is 0.7504m (ADE) and 2.1022m (FDE). As I have suggested, you may want to train with more epochs and starts training the planner at a later stage. Also, you can check the network's output before the motion planner to see if the network's initial plan is good.

LLLiuTC commented 1 year ago

Hi, @MCZhi , thank you for opening source your excellent work. I have trained the validation loss dropped to 0.75m (ADE) and 2.10m (FDE). But my close loop test still can not achieve the paper's performance. uncompressed_scenario_training_20s_training_20s.tfrecord-00106-of-01000.csv This is my close-loop testing result on uncompressed_scenario_training_20s_training_20s.tfrecord-00106-of-01000.: image So, may i know which scenarios you choose to do close loop testing?

MCZhi commented 1 year ago

@LLLiuTC, Hello, please refer to #4.

LLLiuTC commented 1 year ago

thank you very much !