search

chensong1995 / HybridPose

HybridPose: 6D Object Pose Estimation under Hybrid Representation (CVPR 2020)
MIT License
412 stars 64 forks source link

expected evaluation results for ape pretrained weights from linemod? #89

Closed monajalal closed 7 months ago

monajalal commented 9 months ago

Could you please share what number you get for the following commands?

(hybridpose) mona@ada:~/HybridPose$ LD_LIBRARY_PATH=lib/regressor:$LD_LIBRARY_PATH python src/train_core.py --load_dir /home/mona/HybridPose/saved_weights/linemod/ape/checkpoints/0.001/199 --object_name ape
number of model parameters: 12959563
Successfully loaded model from /home/mona/HybridPose/saved_weights/linemod/ape/checkpoints/0.001/199
Testing...
Loss: 0.3184
value of v is:  [1.0010735 3.203219 ]
value of v is:  [0.41729346 0.6314806 ]
value of v is:  [0.7242666  0.42816693]
value of v is:  [0.49726847 2.2989728 ]
value of v is:  [1.1419353  0.44859904]
value of v is:  [1.8812906  0.71737343]
value of v is:  [0.54558986 0.73093534]
value of v is:  [0.93458617 0.44039986]
value of v is:  [0.28327647 0.89215314]
value of v is:  [0.41331306 1.3936875 ]
value of v is:  [0.18460314 0.3257373 ]

ETC

value of v is:  [1.6767471  0.40990227]
value of v is:  [0.37593326 0.7014341 ]
value of v is:  [0.51846135 3.4399436 ]
value of v is:  [0.2994136  0.25986412]
value of v is:  [0.24777637 0.39895052]
value of v is:  [1.1414276  0.34289345]
value of v is:  [0.2974693  0.37998134]
Parameter search for initialization module is terminated.
pose refine search_iter = 0, loss_reduction = -0.000000
pose refine search_iter = 1, loss_reduction = -0.000000
pose refine search_iter = 2, loss_reduction = -0.000000
pose refine search_iter = 3, loss_reduction = -0.000000
pose refine search_iter = 4, loss_reduction = -0.000000
pose refine search_iter = 5, loss_reduction = -0.000000
pose refine search_iter = 6, loss_reduction = -0.000000
pose refine search_iter = 7, loss_reduction = -0.000000
pose refine search_iter = 8, loss_reduction = -0.000000
pose refine search_iter = 9, loss_reduction = -0.000000
pose refine search_iter = 10, loss_reduction = -0.000000
pose refine search_iter = 11, loss_reduction = -0.002232
pose refine search_iter = 12, loss_reduction = -0.001586
pose refine search_iter = 13, loss_reduction = -0.000573

ETC

value of v is:  [0.6803816 2.766172 ]
value of v is:  [0.43708232 0.55195314]
value of v is:  [0.34545782 0.4923641 ]
value of v is:  [1.1943465  0.59530187]
value of v is:  [0.80664915 0.5521057 ]
saved

(hybridpose) mona@ada:~/HybridPose$ python src/evaluate.py
ADD(-S) score of initial prediction is: 0.6266666666666667
ADD(-S) score of final prediction is: 0.6238095238095238

(hybridpose) mona@ada:~/HybridPose/saved_weights/linemod/ape/image/0.001$ ls
total 2.7M
drwxrwxr-x 3 mona mona 4.0K Dec 13 11:07 ..
drwxrwxr-x 2 mona mona 4.0K Dec 13 11:15 .
-rw-rw-r-- 1 mona mona 118K Dec 13 11:42 0_0_sym.jpg
-rw-rw-r-- 1 mona mona 118K Dec 13 11:42 0_0_sym_gt.jpg
-rw-rw-r-- 1 mona mona 6.5K Dec 13 11:42 0_0_mask.jpg
-rw-rw-r-- 1 mona mona 7.2K Dec 13 11:42 0_0_vote_kp_0_pred.jpg
-rw-rw-r-- 1 mona mona 7.3K Dec 13 11:42 0_0_vote_kp_1_pred.jpg
-rw-rw-r-- 1 mona mona 7.2K Dec 13 11:42 0_0_vote_kp_2_pred.jpg
-rw-rw-r-- 1 mona mona 6.9K Dec 13 11:42 0_0_vote_kp_3_pred.jpg
-rw-rw-r-- 1 mona mona 7.2K Dec 13 11:42 0_0_vote_kp_4_pred.jpg
-rw-rw-r-- 1 mona mona 7.1K Dec 13 11:42 0_0_vote_kp_5_pred.jpg
-rw-rw-r-- 1 mona mona 7.3K Dec 13 11:42 0_0_vote_kp_6_pred.jpg
-rw-rw-r-- 1 mona mona 7.2K Dec 13 11:42 0_0_vote_kp_7_pred.jpg
-rw-rw-r-- 1 mona mona 7.3K Dec 13 11:42 0_0_vote_kp_0_gt.jpg
-rw-rw-r-- 1 mona mona 7.4K Dec 13 11:42 0_0_vote_kp_1_gt.jpg
-rw-rw-r-- 1 mona mona 7.3K Dec 13 11:42 0_0_vote_kp_2_gt.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_0_vote_kp_3_gt.jpg
-rw-rw-r-- 1 mona mona 7.3K Dec 13 11:42 0_0_vote_kp_4_gt.jpg
-rw-rw-r-- 1 mona mona 7.1K Dec 13 11:42 0_0_vote_kp_5_gt.jpg
-rw-rw-r-- 1 mona mona 7.3K Dec 13 11:42 0_0_vote_kp_6_gt.jpg
-rw-rw-r-- 1 mona mona 7.2K Dec 13 11:42 0_0_vote_kp_7_gt.jpg
-rw-rw-r-- 1 mona mona 118K Dec 13 11:42 0_0_pts.jpg
-rw-rw-r-- 1 mona mona 118K Dec 13 11:42 0_0_pts_gt.jpg
-rw-rw-r-- 1 mona mona 119K Dec 13 11:42 0_0_gt_graph.jpg
-rw-rw-r-- 1 mona mona 118K Dec 13 11:42 0_0_pred_graph.jpg
-rw-rw-r-- 1 mona mona 121K Dec 13 11:42 0_1_sym.jpg
-rw-rw-r-- 1 mona mona 121K Dec 13 11:42 0_1_sym_gt.jpg
-rw-rw-r-- 1 mona mona 6.4K Dec 13 11:42 0_1_mask.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_1_vote_kp_0_pred.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_1_vote_kp_1_pred.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_1_vote_kp_2_pred.jpg
-rw-rw-r-- 1 mona mona 6.9K Dec 13 11:42 0_1_vote_kp_3_pred.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_1_vote_kp_4_pred.jpg
-rw-rw-r-- 1 mona mona 6.8K Dec 13 11:42 0_1_vote_kp_5_pred.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_1_vote_kp_6_pred.jpg
-rw-rw-r-- 1 mona mona 6.9K Dec 13 11:42 0_1_vote_kp_7_pred.jpg
-rw-rw-r-- 1 mona mona 7.1K Dec 13 11:42 0_1_vote_kp_0_gt.jpg
-rw-rw-r-- 1 mona mona 7.1K Dec 13 11:42 0_1_vote_kp_1_gt.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_1_vote_kp_2_gt.jpg
-rw-rw-r-- 1 mona mona 6.9K Dec 13 11:42 0_1_vote_kp_3_gt.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_1_vote_kp_4_gt.jpg
-rw-rw-r-- 1 mona mona 6.9K Dec 13 11:42 0_1_vote_kp_5_gt.jpg
-rw-rw-r-- 1 mona mona 7.1K Dec 13 11:42 0_1_vote_kp_6_gt.jpg
-rw-rw-r-- 1 mona mona 6.9K Dec 13 11:42 0_1_vote_kp_7_gt.jpg
-rw-rw-r-- 1 mona mona 120K Dec 13 11:42 0_1_pts.jpg
-rw-rw-r-- 1 mona mona 121K Dec 13 11:42 0_1_pts_gt.jpg
-rw-rw-r-- 1 mona mona 121K Dec 13 11:42 0_1_gt_graph.jpg
-rw-rw-r-- 1 mona mona 121K Dec 13 11:42 0_1_pred_graph.jpg
-rw-rw-r-- 1 mona mona 129K Dec 13 11:42 0_2_sym.jpg
-rw-rw-r-- 1 mona mona 129K Dec 13 11:42 0_2_sym_gt.jpg
-rw-rw-r-- 1 mona mona 6.4K Dec 13 11:42 0_2_mask.jpg
-rw-rw-r-- 1 mona mona 6.9K Dec 13 11:42 0_2_vote_kp_0_pred.jpg
-rw-rw-r-- 1 mona mona 6.8K Dec 13 11:42 0_2_vote_kp_1_pred.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_2_vote_kp_2_pred.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_2_vote_kp_3_pred.jpg
-rw-rw-r-- 1 mona mona 6.9K Dec 13 11:42 0_2_vote_kp_4_pred.jpg
-rw-rw-r-- 1 mona mona 6.9K Dec 13 11:42 0_2_vote_kp_5_pred.jpg
-rw-rw-r-- 1 mona mona 6.9K Dec 13 11:42 0_2_vote_kp_6_pred.jpg
-rw-rw-r-- 1 mona mona 6.8K Dec 13 11:42 0_2_vote_kp_7_pred.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_2_vote_kp_0_gt.jpg
-rw-rw-r-- 1 mona mona 6.8K Dec 13 11:42 0_2_vote_kp_1_gt.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_2_vote_kp_2_gt.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_2_vote_kp_3_gt.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_2_vote_kp_4_gt.jpg
-rw-rw-r-- 1 mona mona 6.9K Dec 13 11:42 0_2_vote_kp_5_gt.jpg
-rw-rw-r-- 1 mona mona 7.0K Dec 13 11:42 0_2_vote_kp_6_gt.jpg
-rw-rw-r-- 1 mona mona 6.8K Dec 13 11:42 0_2_vote_kp_7_gt.jpg
-rw-rw-r-- 1 mona mona 129K Dec 13 11:42 0_2_pts.jpg
-rw-rw-r-- 1 mona mona 129K Dec 13 11:42 0_2_pts_gt.jpg
-rw-rw-r-- 1 mona mona 130K Dec 13 11:42 0_2_gt_graph.jpg
-rw-rw-r-- 1 mona mona 129K Dec 13 11:42 0_2_pred_graph.jpg

image

Screenshot from 2023-12-13 13-06-24

Screenshot from 2023-12-13 13-06-40

image

chensong1995 commented 9 months ago

Hi Mona,

Thanks for your question! Please find our evaluation results in Table 1 of the paper. Due to randomness, you may get exactly the same numbers in your local run. It seems like the numbers you have are very close to the numbers reported in the paper, though.

I hope this helps! Let me know if you have further concerns.

monajalal commented 9 months ago

@chensong1995 Thank you so much for your helpful response.

I do agree that for ape, the result of inference on pretrained weight is very close to that reported on Table 1.

However, now, that I trained from scratch on ape class, (already removed the downloaded pretrained weight), I get these much lower scores. Do you know why this is the case?

(hybridpose) mona@ada:~/HybridPose$ python src/evaluate.py
ADD(-S) score of initial prediction is: 0.2419047619047619
ADD(-S) score of final prediction is: 0.22952380952380952

This is how end of the training looked like for me:

Epoch: [198][4/19]  Time: 0.208 (0.281) Sym: 3.0597 (3.6917)    Mask: 0.0075 (0.0082)   Pts: 0.0136 (0.0151)    Graph: 1.3800 (1.6646)  Total: 0.5876 (0.6949)
Epoch: [198][5/19]  Time: 0.311 (0.286) Sym: 3.6039 (3.6771)    Mask: 0.0084 (0.0082)   Pts: 0.0152 (0.0151)    Graph: 2.0709 (1.7323)  Total: 0.7278 (0.7004)
Epoch: [198][6/19]  Time: 0.329 (0.292) Sym: 4.2833 (3.7637)    Mask: 0.0073 (0.0081)   Pts: 0.0149 (0.0151)    Graph: 1.6388 (1.7190)  Total: 0.7483 (0.7072)
Epoch: [198][7/19]  Time: 0.248 (0.287) Sym: 3.7605 (3.7633)    Mask: 0.0067 (0.0079)   Pts: 0.0146 (0.0150)    Graph: 1.7705 (1.7254)  Total: 0.7056 (0.7070)
Epoch: [198][8/19]  Time: 0.342 (0.293) Sym: 3.1455 (3.6946)    Mask: 0.0094 (0.0081)   Pts: 0.0159 (0.0151)    Graph: 2.8550 (1.8509)  Total: 0.7683 (0.7138)
Epoch: [198][9/19]  Time: 0.133 (0.277) Sym: 3.8247 (3.7076)    Mask: 0.0128 (0.0085)   Pts: 0.0183 (0.0154)    Graph: 3.5444 (2.0203)  Total: 0.9323 (0.7357)
Epoch: [198][10/19] Time: 0.282 (0.277) Sym: 3.0792 (3.6505)    Mask: 0.0092 (0.0086)   Pts: 0.0139 (0.0153)    Graph: 1.8583 (2.0055)  Total: 0.6420 (0.7272)
Epoch: [198][11/19] Time: 0.267 (0.276) Sym: 3.8356 (3.6659)    Mask: 0.0092 (0.0087)   Pts: 0.0156 (0.0153)    Graph: 2.0707 (2.0110)  Total: 0.7559 (0.7296)
Epoch: [198][12/19] Time: 0.277 (0.276) Sym: 3.5056 (3.6536)    Mask: 0.0094 (0.0087)   Pts: 0.0134 (0.0152)    Graph: 1.3470 (1.9599)  Total: 0.6289 (0.7218)
Epoch: [198][13/19] Time: 0.217 (0.272) Sym: 3.2381 (3.6239)    Mask: 0.0074 (0.0086)   Pts: 0.0152 (0.0152)    Graph: 1.7904 (1.9478)  Total: 0.6624 (0.7176)
Epoch: [198][14/19] Time: 0.268 (0.272) Sym: 3.8906 (3.6417)    Mask: 0.0062 (0.0085)   Pts: 0.0128 (0.0150)    Graph: 1.3429 (1.9075)  Total: 0.6574 (0.7136)
Epoch: [198][15/19] Time: 0.248 (0.270) Sym: 2.6609 (3.5804)    Mask: 0.0115 (0.0086)   Pts: 0.0150 (0.0150)    Graph: 1.9383 (1.9094)  Total: 0.6217 (0.7078)
Epoch: [198][16/19] Time: 0.265 (0.270) Sym: 4.0085 (3.6056)    Mask: 0.0094 (0.0087)   Pts: 0.0159 (0.0151)    Graph: 2.6993 (1.9559)  Total: 0.8387 (0.7155)
Epoch: [198][17/19] Time: 0.265 (0.270) Sym: 2.8638 (3.5644)    Mask: 0.0082 (0.0087)   Pts: 0.0143 (0.0150)    Graph: 1.7438 (1.9441)  Total: 0.6123 (0.7098)
Epoch: [198][18/19] Time: 0.203 (0.266) Sym: 4.2050 (3.5850)    Mask: 0.0193 (0.0090)   Pts: 0.0156 (0.0150)    Graph: 2.1720 (1.9514)  Total: 0.8126 (0.7131)
Epoch: [199][0/19]  Time: 0.311 (0.311) Sym: 4.2314 (4.2314)    Mask: 0.0100 (0.0100)   Pts: 0.0151 (0.0151)    Graph: 2.2006 (2.2006)  Total: 0.8040 (0.8040)
Epoch: [199][1/19]  Time: 0.314 (0.313) Sym: 3.1559 (3.6936)    Mask: 0.0080 (0.0090)   Pts: 0.0147 (0.0149)    Graph: 1.6411 (1.9208)  Total: 0.6343 (0.7192)
Epoch: [199][2/19]  Time: 0.181 (0.269) Sym: 3.8706 (3.7526)    Mask: 0.0095 (0.0092)   Pts: 0.0182 (0.0160)    Graph: 1.9213 (1.9210)  Total: 0.7710 (0.7364)
Epoch: [199][3/19]  Time: 0.307 (0.279) Sym: 4.3887 (3.9116)    Mask: 0.0115 (0.0098)   Pts: 0.0172 (0.0163)    Graph: 2.0823 (1.9613)  Total: 0.8309 (0.7600)
Epoch: [199][4/19]  Time: 0.371 (0.297) Sym: 3.4709 (3.8235)    Mask: 0.0068 (0.0092)   Pts: 0.0136 (0.0158)    Graph: 1.4313 (1.8553)  Total: 0.6330 (0.7346)
Epoch: [199][5/19]  Time: 0.361 (0.308) Sym: 3.8555 (3.8288)    Mask: 0.0073 (0.0089)   Pts: 0.0174 (0.0160)    Graph: 1.8319 (1.8514)  Total: 0.7499 (0.7372)
Epoch: [199][6/19]  Time: 0.249 (0.299) Sym: 3.7423 (3.8165)    Mask: 0.0074 (0.0086)   Pts: 0.0126 (0.0155)    Graph: 1.3392 (1.7783)  Total: 0.6415 (0.7235)
Epoch: [199][7/19]  Time: 0.208 (0.288) Sym: 3.2162 (3.7414)    Mask: 0.0114 (0.0090)   Pts: 0.0139 (0.0153)    Graph: 1.5887 (1.7546)  Total: 0.6309 (0.7119)
Epoch: [199][8/19]  Time: 0.254 (0.284) Sym: 3.3017 (3.6926)    Mask: 0.0106 (0.0092)   Pts: 0.0149 (0.0153)    Graph: 1.5807 (1.7353)  Total: 0.6481 (0.7048)
Epoch: [199][9/19]  Time: 0.316 (0.287) Sym: 2.5885 (3.5822)    Mask: 0.0076 (0.0090)   Pts: 0.0153 (0.0153)    Graph: 1.6329 (1.7250)  Total: 0.5825 (0.6926)
Epoch: [199][10/19] Time: 0.367 (0.295) Sym: 3.6529 (3.5886)    Mask: 0.0075 (0.0089)   Pts: 0.0139 (0.0152)    Graph: 1.8202 (1.7337)  Total: 0.6942 (0.6927)
Epoch: [199][11/19] Time: 0.240 (0.290) Sym: 3.4518 (3.5772)    Mask: 0.0101 (0.0090)   Pts: 0.0138 (0.0150)    Graph: 1.9123 (1.7486)  Total: 0.6842 (0.6920)
Epoch: [199][12/19] Time: 0.213 (0.284) Sym: 4.0188 (3.6112)    Mask: 0.0084 (0.0089)   Pts: 0.0170 (0.0152)    Graph: 2.4308 (1.8010)  Total: 0.8234 (0.7021)
Epoch: [199][13/19] Time: 0.358 (0.289) Sym: 4.3164 (3.6615)    Mask: 0.0069 (0.0088)   Pts: 0.0134 (0.0151)    Graph: 2.0667 (1.8200)  Total: 0.7788 (0.7076)
Epoch: [199][14/19] Time: 0.184 (0.282) Sym: 3.8386 (3.6733)    Mask: 0.0088 (0.0088)   Pts: 0.0154 (0.0151)    Graph: 2.3918 (1.8581)  Total: 0.7863 (0.7129)
Epoch: [199][15/19] Time: 0.282 (0.282) Sym: 3.2402 (3.6463)    Mask: 0.0082 (0.0087)   Pts: 0.0155 (0.0151)    Graph: 1.7861 (1.8536)  Total: 0.6656 (0.7099)
Epoch: [199][16/19] Time: 0.236 (0.280) Sym: 3.5804 (3.6424)    Mask: 0.0078 (0.0087)   Pts: 0.0146 (0.0151)    Graph: 1.5218 (1.8341)  Total: 0.6642 (0.7072)
Epoch: [199][17/19] Time: 0.268 (0.279) Sym: 3.3273 (3.6249)    Mask: 0.0099 (0.0088)   Pts: 0.0150 (0.0151)    Graph: 2.3768 (1.8643)  Total: 0.7301 (0.7085)
Epoch: [199][18/19] Time: 0.245 (0.277) Sym: 4.5181 (3.6537)    Mask: 0.0079 (0.0087)   Pts: 0.0143 (0.0151)    Graph: 2.1356 (1.8730)  Total: 0.8167 (0.7120)
Testing...
Loss: 0.6049
Successfully saved model into saved_weights/linemod/ape/checkpoints/0.001/199
value of v is:  [0.96671426 0.43658534]
value of v is:  [0.51299405 1.6662292 ]
value of v is:  [1.4882765  0.97056437]
value of v is:  [1.080338  4.6441827]
value of v is:  [0.9764906 0.626466 ]
chensong1995 commented 9 months ago

Hi Mona,

Thanks for your follow-up! I suspect this is because you did not include the blender and fuse data during training. In another thread, you were recommended to remove them for the purpose of evaluation. However, for training, you do need to spend the time to properly configure these two datasets.

I hope this helps! Let me know if you have any further concerns.