woven-planet / l5kit

L5Kit - https://woven.toyota
https://woven-planet.github.io/l5kit
857 stars 278 forks source link

Reproducing ADE score in SimNet paper #369

Open pjw1 opened 2 years ago

pjw1 commented 2 years ago

Hi! Thank you for kindly sharing this code.

I wonder if I can reproduce the results (ADE score) in SimNet paper. The paper says the ADE is 0.49, 0.76, 1.32 at 0.5, 1, 2sec respectively. image

But the pre-trained simulation model can't reproduce the score and the gap is quite huge. (https://lyft-l5-datasets-public.s3-us-west-2.amazonaws.com/models/simulation_models/simulation_model_20210416_5steps.pt) 0.96, 1.42, 1.64 at 0.5, 1, 2.

Can you share the model used in the paper or implementation details to reproduce the result?

Thank you in advance!