Open taeyeopl opened 3 years ago
I used the provided checkpoints vgg16_ycb_video_epoch_16.checkpoint.pth to check the YCB dataset(ycb_video_keyframe) performance. I got the below results. I have some questions related to the results.
Q1. Do pretrained weights show same pretrained weights of the paper PoseCNN and DexYCB?? If not, can you share your results based on all the pretrained results of all the datasets(ycb_video, dex_ycb, ycb_object)?? I only observed that the YCB dataset performance except other datasets (dex_ycb_s0~3, ycb_object).
* ADD: 0.537 | [[Provided checkpoints](https://drive.google.com/file/d/1-ECAkkTRfa1jJ9YBTzf04wxCGw6-m5d4/view) 0.536727] * ADD-S: 0.759 | [[Provided checkpoints](https://drive.google.com/file/d/1-ECAkkTRfa1jJ9YBTzf04wxCGw6-m5d4/view) 0.743643] * ADD refined: 0.793 | [[Provided checkpoints](https://drive.google.com/file/d/1-ECAkkTRfa1jJ9YBTzf04wxCGw6-m5d4/view) 0.075520] * ADD-S refined: 0.993 | [[Provided checkpoints](https://drive.google.com/file/d/1-ECAkkTRfa1jJ9YBTzf04wxCGw6-m5d4/view) 0.119003]
Q2. I observed that refined performance was quite lower than reported. I felt it is wired. Can you explain why??
time ./tools/test_net.py --gpu 0 \ --network posecnn \ --pretrained data/checkpoints/ycb_video/vgg16_ycb_video_epoch_16.checkpoint.pth \ --dataset ycb_video_keyframe \ --cfg experiments/cfgs/ycb_video.yml ==================My Performance(ALL)====================== ADD: 0.536727 ADD-S: 0.743643 ADD refined: 0.075520 ADD-S refined: 0.119003 ==================ADD====================== all: 0.536727 002_master_chef_can: 0.615168 003_cracker_box: 0.617424 004_sugar_box: 0.532164 005_tomato_soup_can: 0.708913 006_mustard_bottle: 0.797891 007_tuna_fish_can: 0.674004 008_pudding_box: 0.479661 009_gelatin_box: 0.770318 010_potted_meat_can: 0.688317 011_banana: 0.708651 019_pitcher_base: 0.723790 021_bleach_cleanser: 0.473929 024_bowl: 0.062281 025_mug: 0.534518 035_power_drill: 0.547650 036_wood_block: 0.023127 037_scissors: 0.465419 040_large_marker: 0.591836 051_large_clamp: 0.109451 052_extra_large_clamp: 0.032313 061_foam_brick: 0.701915 0.615168 0.617424 0.532164 0.708913 0.797891 0.674004 0.479661 0.770318 0.688317 0.708651 0.723790 0.473929 0.062281 0.534518 0.547650 0.023127 0.465419 0.591836 0.109451 0.032313 0.701915 0.536727 ycb_video =========================================== ==================ADD-S==================== all: 0.743643 002_master_chef_can: 0.877846 003_cracker_box: 0.803593 004_sugar_box: 0.741124 005_tomato_soup_can: 0.838784 006_mustard_bottle: 0.900914 007_tuna_fish_can: 0.874633 008_pudding_box: 0.700458 009_gelatin_box: 0.885764 010_potted_meat_can: 0.859981 011_banana: 0.868941 019_pitcher_base: 0.862067 021_bleach_cleanser: 0.654374 024_bowl: 0.730333 025_mug: 0.761190 035_power_drill: 0.748332 036_wood_block: 0.257070 037_scissors: 0.631964 040_large_marker: 0.718532 051_large_clamp: 0.412216 052_extra_large_clamp: 0.308376 061_foam_brick: 0.851005 0.877846 0.803593 0.741124 0.838784 0.900914 0.874633 0.700458 0.885764 0.859981 0.868941 0.862067 0.654374 0.730333 0.761190 0.748332 0.257070 0.631964 0.718532 0.412216 0.308376 0.851005 0.743643 ycb_video =========================================== ==================ADD refined====================== all: 0.075520 002_master_chef_can: 0.123327 003_cracker_box: 0.046552 004_sugar_box: 0.077753 005_tomato_soup_can: 0.065748 006_mustard_bottle: 0.079828 007_tuna_fish_can: 0.135382 008_pudding_box: 0.047530 009_gelatin_box: 0.047532 010_potted_meat_can: 0.088641 011_banana: 0.081146 019_pitcher_base: 0.097756 021_bleach_cleanser: 0.041396 024_bowl: 0.008421 025_mug: 0.172956 035_power_drill: 0.085137 036_wood_block: 0.040056 037_scissors: 0.046730 040_large_marker: 0.060406 051_large_clamp: 0.048245 052_extra_large_clamp: 0.021109 061_foam_brick: 0.058939 0.123327 0.046552 0.077753 0.065748 0.079828 0.135382 0.047530 0.047532 0.088641 0.081146 0.097756 0.041396 0.008421 0.172956 0.085137 0.040056 0.046730 0.060406 0.048245 0.021109 0.058939 0.075520 ycb_video =========================================== ==================ADD-S refined==================== all: 0.119003 002_master_chef_can: 0.173956 003_cracker_box: 0.081636 004_sugar_box: 0.107987 005_tomato_soup_can: 0.092742 006_mustard_bottle: 0.108868 007_tuna_fish_can: 0.191927 008_pudding_box: 0.069155 009_gelatin_box: 0.058518 010_potted_meat_can: 0.130085 011_banana: 0.119868 019_pitcher_base: 0.124502 021_bleach_cleanser: 0.060756 024_bowl: 0.074100 025_mug: 0.189300 035_power_drill: 0.123411 036_wood_block: 0.179038 037_scissors: 0.056125 040_large_marker: 0.081108 051_large_clamp: 0.112593 052_extra_large_clamp: 0.181822 061_foam_brick: 0.088207 0.173956 0.081636 0.107987 0.092742 0.108868 0.191927 0.069155 0.058518 0.130085 0.119868 0.124502 0.060756 0.074100 0.189300 0.123411 0.179038 0.056125 0.081108 0.112593 0.181822 0.088207 0.119003 ycb_video =========================================== Process finished with exit code 0
Hello Lee, I find that you have test the checkpoint, so i'd like to seek some help from you. My own checkpoint(obtained from training code) get worse performance(ADD = 51,ADD-S = 73 after refinement) than paper report(ADD = 79, ADD-S = 93 after refinement). I'd like to ask you that whether you meet the same situation with me. Thank you! @yuxng I'll be very appreciate if yu xiang can offer some help
I used the provided checkpoints vgg16_ycb_video_epoch_16.checkpoint.pth to check the YCB dataset(ycb_video_keyframe) performance. I got the below results. I have some questions related to the results.
Q1. Do pretrained weights show same pretrained weights of the paper PoseCNN and DexYCB?? If not, can you share your results based on all the pretrained results of all the datasets(ycb_video, dex_ycb, ycb_object)?? I only observed that the YCB dataset performance except other datasets (dex_ycb_s0~3, ycb_object).
Q2. I observed that refined performance was quite lower than reported. I felt it is wired. Can you explain why??