metrics show as zero during evaluation on linemod cat using your pretrained weights

[monajalal]

TLDR: could you please share the results of running the following command with and without test.vsd True flag? I am not sure what the results should look like exactly.

following https://github.com/zju3dv/clean-pvnet/issues/49#issuecomment-603067491 and setting test.vsd to True, I get:

(clean-pvnet) mona@ada:~/clean-pvnet$ python run.py --type evaluate --cfg_file configs/linemod.yaml model cat cls_type cat  test.vsd True
Load model: data/model/pvnet/cat/199.pth
loading annotations into memory...
Done (t=0.19s)
creating index...
index created!
loading annotations into memory...
Done (t=0.03s)
creating index...
index created!
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1002/1002 [00:35<00:00, 28.08it/s]
2d projections metric: 0.0
ADD metric: 0.0
5 cm 5 degree metric: 0.0
mask ap70: 0.9960079840319361

also, if I don't use that argument, I get this.

(clean-pvnet) mona@ada:~/clean-pvnet$ python run.py --type evaluate --cfg_file configs/linemod.yaml model cat cls_type cat  
Load model: data/model/pvnet/cat/199.pth
loading annotations into memory...
Done (t=0.12s)
creating index...
index created!
loading annotations into memory...
Done (t=0.03s)
creating index...
index created!
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1002/1002 [00:35<00:00, 28.20it/s]
2d projections metric: 0.0
ADD metric: 0.0
5 cm 5 degree metric: 0.0
mask ap70: 0.9960079840319361

[ZYLer0510]

Hi, have you solved this problem? I'm having the same problem.

[monajalal]

@ZYLer0510 no.

[ZYLer0510]

@monajalal I think you can try: ./lib/networks/pvnet/resnet18.py#decode_keypoint()：ransac_voting_layer_v3() -> ransac_voting_layer() for example: # kpt_2d = ransac_voting_layer_v3(mask, vertex, 128, inlier_thresh=0.99, max_num=100) kpt_2d = ransac_voting_layer(mask, vertex, 128, inlier_thresh=0.99, max_num=100)

[monajalal]

@ZYLer0510 could you please share why you switched from ransac_voting_layer_v3 to ransac_voting_layer? Thank you. Also, how were you able to figure that this was the cause of problem?

[monajalal]

@ZYLer0510 did you also change mean = ransac_voting_layer_v3(mask, vertex, 512, inlier_thresh=0.99)

to

mean = ransac_voting_layer(mask, vertex, 512, inlier_thresh=0.99)

?

[monajalal]

@ZYLer0510

doing what you suggested, I got an error

(clean-pvnet) mona@ada:~/clean-pvnet$ python run.py --type evaluate --cfg_file configs/linemod.yaml model cat cls_type cat  test.vsd True
Load model: data/model/pvnet/cat/199.pth
loading annotations into memory...
Done (t=0.12s)
creating index...
index created!
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
  0%|                                                                                                                                                                              | 0/1002 [00:00<?, ?it/s]type(cur_mask):  <class 'torch.Tensor'>
  0%|                                                                                                                                                                              | 0/1002 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/mona/clean-pvnet/run.py", line 276, in <module>
    globals()['run_'+args.type]()
  File "/home/mona/clean-pvnet/run.py", line 78, in run_evaluate
    output = network(inp)
  File "/home/mona/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/mona/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/mona/clean-pvnet/lib/networks/pvnet/resnet18.py", line 101, in forward
    self.decode_keypoint(ret)
  File "/home/mona/clean-pvnet/lib/networks/pvnet/resnet18.py", line 76, in decode_keypoint
    kpt_2d = ransac_voting_layer(mask, vertex, 128, inlier_thresh=0.99, max_num=100)
  File "/home/mona/clean-pvnet/lib/csrc/ransac_voting/ransac_voting_gpu.py", line 38, in ransac_voting_layer
    direct = vertex[bi].masked_select(torch.unsqueeze(torch.unsqueeze(cur_mask, 2), 3))  # [tn,vn,2]
RuntimeError: masked_select: expected BoolTensor for mask
(clean-pvnet) mona@ada:~/clean-pvnet$ git diff
diff --git a/.gitignore b/.gitignore
index afbb944..dd3b9bc 100644
--- a/.gitignore
+++ b/.gitignore
@@ -10,4 +10,4 @@ __pycache__/
 *.jpg
 *.ipynb
 *.npy
-
+cat_backup_training_linemod
diff --git a/lib/networks/pvnet/resnet18.py b/lib/networks/pvnet/resnet18.py
index 7fddeb3..f89b781 100644
--- a/lib/networks/pvnet/resnet18.py
+++ b/lib/networks/pvnet/resnet18.py
@@ -72,7 +72,8 @@ class Resnet18(nn.Module):
             kpt_2d, var = estimate_voting_distribution_with_mean(mask, vertex, mean)
             output.update({'mask': mask, 'kpt_2d': kpt_2d, 'var': var})
         else:
-            kpt_2d = ransac_voting_layer_v3(mask, vertex, 128, inlier_thresh=0.99, max_num=100)
+            #kpt_2d = ransac_voting_layer_v3(mask, vertex, 128, inlier_thresh=0.99, max_num=100) #original
+            kpt_2d = ransac_voting_layer(mask, vertex, 128, inlier_thresh=0.99, max_num=100)
             output.update({'mask': mask, 'kpt_2d': kpt_2d})

     def forward(self, x, feature_alignment=False):

[ZYLer0510]

@ZYLer0510 did you also change mean = ransac_voting_layer_v3(mask, vertex, 512, inlier_thresh=0.99)

to

mean = ransac_voting_layer(mask, vertex, 512, inlier_thresh=0.99)

?

I haven't tried un-PnP, you could try it. I'm just starting to look at this code.

[ZYLer0510]

@ZYLer0510

doing what you suggested, I got an error

(clean-pvnet) mona@ada:~/clean-pvnet$ python run.py --type evaluate --cfg_file configs/linemod.yaml model cat cls_type cat  test.vsd True
Load model: data/model/pvnet/cat/199.pth
loading annotations into memory...
Done (t=0.12s)
creating index...
index created!
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
  0%|                                                                                                                                                                              | 0/1002 [00:00<?, ?it/s]type(cur_mask):  <class 'torch.Tensor'>
  0%|                                                                                                                                                                              | 0/1002 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/mona/clean-pvnet/run.py", line 276, in <module>
    globals()['run_'+args.type]()
  File "/home/mona/clean-pvnet/run.py", line 78, in run_evaluate
    output = network(inp)
  File "/home/mona/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/mona/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/mona/clean-pvnet/lib/networks/pvnet/resnet18.py", line 101, in forward
    self.decode_keypoint(ret)
  File "/home/mona/clean-pvnet/lib/networks/pvnet/resnet18.py", line 76, in decode_keypoint
    kpt_2d = ransac_voting_layer(mask, vertex, 128, inlier_thresh=0.99, max_num=100)
  File "/home/mona/clean-pvnet/lib/csrc/ransac_voting/ransac_voting_gpu.py", line 38, in ransac_voting_layer
    direct = vertex[bi].masked_select(torch.unsqueeze(torch.unsqueeze(cur_mask, 2), 3))  # [tn,vn,2]
RuntimeError: masked_select: expected BoolTensor for mask
(clean-pvnet) mona@ada:~/clean-pvnet$ git diff
diff --git a/.gitignore b/.gitignore
index afbb944..dd3b9bc 100644
--- a/.gitignore
+++ b/.gitignore
@@ -10,4 +10,4 @@ __pycache__/
 *.jpg
 *.ipynb
 *.npy
-
+cat_backup_training_linemod
diff --git a/lib/networks/pvnet/resnet18.py b/lib/networks/pvnet/resnet18.py
index 7fddeb3..f89b781 100644
--- a/lib/networks/pvnet/resnet18.py
+++ b/lib/networks/pvnet/resnet18.py
@@ -72,7 +72,8 @@ class Resnet18(nn.Module):
             kpt_2d, var = estimate_voting_distribution_with_mean(mask, vertex, mean)
             output.update({'mask': mask, 'kpt_2d': kpt_2d, 'var': var})
         else:
-            kpt_2d = ransac_voting_layer_v3(mask, vertex, 128, inlier_thresh=0.99, max_num=100)
+            #kpt_2d = ransac_voting_layer_v3(mask, vertex, 128, inlier_thresh=0.99, max_num=100) #original
+            kpt_2d = ransac_voting_layer(mask, vertex, 128, inlier_thresh=0.99, max_num=100)
             output.update({'mask': mask, 'kpt_2d': kpt_2d})

     def forward(self, x, feature_alignment=False):

RuntimeError: masked_select: expected BoolTensor for mask You could add cur_mask = cur_mask.to(torch.bool) before direct = vertex[bi].masked_select(torch.unsqueeze(torch.unsqueeze(cur_mask, 2), 3))

[ZYLer0510]

@ZYLer0510 could you please share why you switched from ransac_voting_layer_v3 to ransac_voting_layer? Thank you. Also, how were you able to figure that this was the cause of problem?

Sorry, I don't know exactly why, I just tried to switch it and found that it works, but I haven't found the exact reason yet. It's already one o'clock here (China), and I'll see this code tomorrow.

[monajalal]

@ZYLer0510

Thank you so much for your response. It worked.

Were you able to create a synthetic dataset and pass it to clean-pvnet? Would you please be able to update on it when you get a chance?

Have a nice day.

type(cur_mask):  <class 'torch.Tensor'>
type(cur_mask):  <class 'torch.Tensor'>
type(cur_mask):  <class 'torch.Tensor'>
type(cur_mask):  <class 'torch.Tensor'>
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 1001/1002 [00:36<00:00, 31.37it/s]type(cur_mask):  <class 'torch.Tensor'>
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1002/1002 [00:36<00:00, 27.12it/s]
2d projections metric: 0.9970059880239521
ADD metric: 0.7754491017964071
5 cm 5 degree metric: 0.9830339321357285
mask ap70: 0.9960079840319361

[ZhuangTingT]

@ZYLer0510

Thank you so much for your response. It worked.

Were you able to create a synthetic dataset and pass it to clean-pvnet? Would you please be able to update on it when you get a chance?

Have a nice day.

type(cur_mask):  <class 'torch.Tensor'>
type(cur_mask):  <class 'torch.Tensor'>
type(cur_mask):  <class 'torch.Tensor'>
type(cur_mask):  <class 'torch.Tensor'>
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 1001/1002 [00:36<00:00, 31.37it/s]type(cur_mask):  <class 'torch.Tensor'>
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1002/1002 [00:36<00:00, 27.12it/s]
2d projections metric: 0.9970059880239521
ADD metric: 0.7754491017964071
5 cm 5 degree metric: 0.9830339321357285
mask ap70: 0.9960079840319361

hi, may you tell me your version of torch, python?

i have changed the code as above, but the test result of ADD metric is still 0.09......