[Question] comparing the results

[limaolin2017]

I would like to compare the results before applying CAMM and after applying CAMM with the ground truth.

I found that there are 300 mesh-.obj files in the iiwa dataset but only 299 in '/camm/logdir/iiwa-test1/iiwa-ft2-rendering/'. Additionally, '/camm/logdir/iiwa-test1/iiwa-skel-rendering/' has only 299 iiwa00-mesh-.obj files. How can I compare?

Assumptions:

1. The real value is in the iiwa dataset zip file.
2. The directory before applying CAMM is '/camm/logdir/iiwa-test1/iiwa-ft2-rendering/'.
3. The directory after applying CAMM is '/camm/logdir/iiwa-test1/iiwa-skel-rendering/'.

[kts707]

When the results are saved to directories like logdir/iiwa-test1/iiwa-ft2-rendering/ or logdir/iiwa-test1/iiwa-skel-rendering/, the index for the filenames starts from 0 (0 to 299 for iiwa). For the meshes in my dataset folder, the index starts from 1 (1 to 300). Let me know if indeed only 299 meshes are saved and I will look into it. Thanks!

To run quantitative evaluation on the results, simply follow the evaluation script for iiwa dataset. Results will be saved under iiwa_eval/. You can replace the results dir (second command line argument) with either logdir/iiwa-test1/iiwa-ft2-rendering/ or logdir/iiwa-test1/iiwa-skel-rendering to run the evaluation for the desired experiment setup.

In the case where you want to compare the results with and without kinematic chain, you might want to change where numbers are saved in the evalutaion bash script because currently it will always save them to the same folder and replace the results from previous evaluation.

[limaolin2017]

Hi! Thank you for your reply!

For directories like logdir/iiwa-test1/iiwa-ft2-rendering/ or logdir/iiwa-test1/iiwa-skel-rendering/ It's true that only 299 meshes were saved, I attached a preview of that.

[kts707]

Thanks for checking!

Could you check if only 299 images are saved under DAVIS/JPEGImages/? If so, it is likely due to the last video frame not being saved during the data preprocessing.

For evaluation and comparision purpose, I think using 299 frames would be fine for now (the bash script for evaluation will read the frames in sequence, so it would evaluate only based on the 299 frames). I am currently busy with some other tasks... will look at it a bit later. Thanks again for finding this!

[limaolin2017]

Yes, I check DAVIS/JPEGImages/, only 299 images. For evaluation and comparison purpose, you are right, 299 images are enough. Its fine, thank you for your reply!

[kts707]

Thanks for checking and letting me know! I will try to find a time to look into it as soon as possible. For now, I think missing 1 out of 300 frame would not change the results by too far in terms of evaluation and comparison.

[kts707]

Hi, I tested the data preprocessing on my end. I am able to get 300 images for each sequence under DAVIS/JPEGImages/. The index is going from 00000.png to 00299.png. I am not sure where the discrepancy comes from... (might be from different ffmpeg versions?)

This difference is very small if only the last video frame is not saved by the processing script. They would not affect the computation of optical flow, DINO features, and masks. I think everything else should still run smoothly. So this will have very minimal on the results because the robot arm's ending pose is the same as its starting pose in my videos.