search

m-hasan-n / pooling

"Pooling Toolbox" is the code of our work "Maneuver-Aware Pooling for Vehicle Trajectory Prediction".
MIT License
50 stars 8 forks source link

The measuring unit of MSE in utils.py #2

Closed VioletPepper closed 3 years ago

VioletPepper commented 3 years ago

Hi, I wonder the unit of Mean Squared Error when it comes to the polar coordinates. I think the unit is not feet^2 or meter^2 anymore, but the evaluation metric reported in the paper is RMSE in meter, which kind of confused me. Did you convert the polar coordinates into Euclidean coordinates when calculating RMSE, or just report the results in polar coordiantes?

Thanks!

m-hasan-n commented 3 years ago

Hi, thanks for your interest in our work. Polar coordinates are represented by two parameters: radius (in feet) and angle (in radians). Since a radian represents the ratio of arc length to radius length, it can be considered as a "dimensionless" quantity. Thus, the resulting error is still in feet and can be converted to meters as reported in the paper.

VioletPepper commented 3 years ago

Thanks for your reply! Though the unit here is still meter, the meaning of RMSE in polar coordinates is different from the previous work in Euclidean coordinates (S-LSTM, CSP and S-GAN). For fair comparison, all the baselines and the proposed method should use the same coordinates to calculte RMSE, right? Were all the reported RMSE results in the paper calculated in polar coodinates?

m-hasan-n commented 3 years ago

There are two points to think of: the evaluation criteria and the unit. RMSE represents the error between two quantities regardless of how they are represented. Whether Euclidean or Polar, RMSE finds the difference between the groundtruth and predicted trajectories. Regarding the unit, it is the same for all approaches. Hence, I believe that RMSE in meters works as a fair metric to compare the baselines. Please note that the main difference in the proposed method is changing the trajectory representation from Euclidean to Polar, and we compared the baselines accordingly.

VioletPepper commented 3 years ago

Got it! Thank you very much for the timely answer.

guoyage commented 5 months ago

There are two points to think of: the evaluation criteria and the unit. RMSE represents the error between two quantities regardless of how they are represented. Whether Euclidean or Polar, RMSE finds the difference between the groundtruth and predicted trajectories. Regarding the unit, it is the same for all approaches. Hence, I believe that RMSE in meters works as a fair metric to compare the baselines. Please note that the main difference in the proposed method is changing the trajectory representation from Euclidean to Polar, and we compared the baselines accordingly.

There are two points to think of: the evaluation criteria and the unit. RMSE represents the error between two quantities regardless of how they are represented. Whether Euclidean or Polar, RMSE finds the difference between the groundtruth and predicted trajectories. Regarding the unit, it is the same for all approaches. Hence, I believe that RMSE in meters works as a fair metric to compare the baselines. Please note that the main difference in the proposed method is changing the trajectory representation from Euclidean to Polar, and we compared the baselines accordingly.

Hello, I have the same confusion about RMSE. First of all, the RMSE calculation of the Polar model in the code is performed in the polar coordinate system, which brings about the first problem: in the polar coordinate system, the dimensions of radius and angle are not the same, and the two dimensions are different. Is it reasonable to add the values directly? Secondly, I tried to convert the predicted trajectory from the polar coordinate system to the Cartesian coordinate system. At this time, I found that the RMSE in the Cartesian coordinate system would increase, which brought about the second confusion: the difference between Polar and Cartesian coordinates. The Polar-V model uses a polar coordinate system, while Slstm and CSP use a Cartesian coordinate system. Is the RMSE comparison fair at this time? Very much looking forward to your reply.