search

ir-lab / alpha-MDF

Codebase for α-MDF at CoRL 2023
MIT License
8 stars 2 forks source link

For the evaluations on UR5 manipulation, how to calculate the MAE in table 2? #4

Open surprisesoon opened 6 months ago

surprisesoon commented 6 months ago

I've tried to calculate the MAE using the code below. However, the result is very different from table 2 in the paper. https://github.com/ir-lab/alpha-MDF/blob/c5d6cf4fc7164200dbb4963c9e14dead78ac431b/latent_space/engine_UR5.py#L471

If the 'state' obtained by online_test in engine_UR5.py(i.e. the code cited above) cannot be directly used to calculate MAE, do other data processing need to be performed before calculating MAE?

In addition, is MAE obtained by using all joint angles？

liuxiao1468 commented 6 months ago

The MAE I reported in the paper is calculated in End-Effector space from the 3D distance (cm) from the predicted end-effector (x,y,z) to the ground truth (x,y,z), and in the joint angle (deg) space. I normalized the state in training so the predictions need to be unnormalized so they are in the correct scale.

Here is how I gathered the dataset https://github.com/ir-lab/alpha-MDF/blob/main/latent_space/dataset/make_dataset_mujoco.py

The accuracy will for sure be different if we use different robot configurations, let me know if you have more questions.