Training on own dataset not fully successful

[krullgit]

Hi, thanks for the repo and paper, seriously good work! I trained the network on my own data and maybe you have an idea where to look to improve performance, judged on these results. Especially ADD and ARP are super low. Any idea? I suspect that I probably did something wrong with data preparation since the training on the LM data works great.

[0708 15:57:00 @eval.py:75] ------------ x ----------- [0708 15:57:00 @eval.py:76] [rot_thresh, trans_thresh: RotAcc, TraAcc, SpcAcc [0708 15:57:00 @eval.py:77] average_accuracy[-1, -1.00]: 74.14, 80.30, 69.60 [0708 15:57:00 @eval.py:84] average_accuracy[ 2, 0.02]: 48.48, 70.71, 43.43 [0708 15:57:00 @eval.py:84] average_accuracy[ 5, 0.05]: 82.83, 83.84, 78.79 [0708 15:57:00 @eval.py:84] average_accuracy[10, 0.10]: 90.91, 91.92, 88.89

[0708 15:57:00 @eval.py:92] ---------- performance over 1 classes ----------- [0708 15:57:00 @eval.py:93] [rot_thresh, trans_thresh: RotAcc, TraAcc, SpcAcc [0708 15:57:00 @eval.py:95] average_accuracy[-1, -1.00]: 74.14, 80.30, 69.60 [0708 15:57:00 @eval.py:104] average_accuracy[ 2, 0.02]: 48.48, 70.71, 43.43 [0708 15:57:00 @eval.py:104] average_accuracy[ 5, 0.05]: 82.83, 83.84, 78.79 [0708 15:57:00 @eval.py:104] average_accuracy[10, 0.10]: 90.91, 91.92, 88.89

[0708 15:57:00 @eval.py:120]

• ---------------------------------------------------------------------------------------------------- * Metric ADD
• ---------------------------------------------------------------------------------------------------- * [0708 15:57:04 @eval.py:174] x [0708 15:57:04 @eval.py:197] threshold=[0.0, 0.10], area: 14.62 [0708 15:57:04 @eval.py:198] threshold=0.02, correct poses: 0.0, all poses: 99.0, accuracy: 0.00 [0708 15:57:04 @eval.py:202] threshold=0.05, correct poses: 9.0, all poses: 99.0, accuracy: 9.09 [0708 15:57:04 @eval.py:206] threshold=0.10, correct poses: 45.0, all poses: 99.0, accuracy: 45.45 [0708 15:57:04 @eval.py:210]
  [0708 15:57:04 @eval.py:215] ============================== [0708 15:57:04 @eval.py:216]
  [0708 15:57:04 @eval.py:219] ---------- add performance over 1 classes ----------- [0708 15:57:04 @eval.py:220] iter 1 [0708 15:57:04 @eval.py:221] threshold=[0.0, 0.10], area: 14.62 [0708 15:57:04 @eval.py:223] threshold=0.02, mean accuracy: 0.00 [0708 15:57:04 @eval.py:225] threshold=0.05, mean accuracy: 9.09 [0708 15:57:04 @eval.py:227] threshold=0.10, mean accuracy: 45.45 [0708 15:57:04 @eval.py:229]
  [0708 15:57:04 @eval.py:230] ============================== [0708 15:57:04 @eval.py:240]
• ---------------------------------------------------------------------------------------------------- * Metric ARP_2D (Average Re-Projection 2D)
• ---------------------------------------------------------------------------------------------------- * [0708 15:57:11 @eval.py:300] x [0708 15:57:11 @eval.py:327] threshold=[0, 50], area: 0.00 [0708 15:57:11 @eval.py:328] threshold=2, correct poses: 0.0, all poses: 99.0, accuracy: 0.00 [0708 15:57:11 @eval.py:330] threshold=5, correct poses: 0.0, all poses: 99.0, accuracy: 0.00 [0708 15:57:11 @eval.py:332] threshold=10, correct poses: 0.0, all poses: 99.0, accuracy: 0.00 [0708 15:57:11 @eval.py:335] threshold=20, correct poses: 0.0, all poses: 99.0, accuracy: 0.00 [0708 15:57:11 @eval.py:338]
  [0708 15:57:11 @eval.py:342] ============================== [0708 15:57:11 @eval.py:343]
  [0708 15:57:11 @eval.py:346] ---------- arp 2d performance over 1 classes ----------- [0708 15:57:11 @eval.py:347] iter 1 [0708 15:57:11 @eval.py:348] threshold=[0, 50], area: 0.00 [0708 15:57:11 @eval.py:350] threshold=2, mean accuracy: 0.00 [0708 15:57:11 @eval.py:352] threshold=5, mean accuracy: 0.00 [0708 15:57:11 @eval.py:354] threshold=10, mean accuracy: 0.00 [0708 15:57:11 @eval.py:356] threshold=20, mean accuracy: 0.00 [0708 15:57:11 @eval.py:358]
  [0708 15:57:11 @eval.py:359] ==============================

[Lakonik]

Hi! Thank you for your interest in our work.

I think it is possibly related to data preparation, OR, perhaps your data is vastly different from LineMOD so the hyperparameters or evaluation metrics might not work well. For example, if your objects have a much larger scale or are very distant, then the current evaluation metrics might be too strict.

[krullgit]

Thanks! I found a problem in my coord files and ADD looks good now. However, ARP_2d is still zero (threshold=20, correct poses: 1.0, all poses: 99.0, accuracy: 1.01). Can it have something to do with the norm_factor which doesn't go up? Here is, how it looks right now (orange is LM training and grey is mine):

[Lakonik]

I think this norm_factor is OK. It's rough measure of how concentrated the predicted pose distributions are, so lower norm_factor indicates higher uncertainty, possibly because your dataset is more challenging.

[petrock99]

@krullgit, how did you generate your own dataset? Specifically, how did you generate the coor.pkl files that the algorithm uses as the ground-truth for each object? Or did you distill it down somehow to what the algorithm actually needs? I'm having trouble generating them for my own dataset. Thanks.

[krullgit]

The coor pkl files contain the point cloud of the object itself :) So assuming you have an image with the according depth per pixel and a segmentation map, you first have to mask the depth map with the segmentation map. Now, you have the depth where the object is. Next you have to normalize all the remaining depth values (x,y,z) with the center of the object. Or in other words, you substract the center point from all other 3d points. Now, every point from the object is given in relation to the center of that object. Thats the pkl file. Hope that helps

[guan2000910]

@krullgit, Hello, I would like to run the code for this project on an industrial dataset, can you provide any thoughts on getting your own dataset to run successfully? And exactly how did you implement the generation of the pkl file? Thank you!