Error in training process with validation check

SylarZiling commented 1 year ago

Hi I am running into some problem when i am trying the training process. my env: ubuntu 20.04 & 18.04 I tried both environment and both has the same error at validation sanity check process the direct output is ''' in validation_epoch_end self.logger.experiment.addscalar(f'metrics{valset_idx}/{k}', v, global_step=cur_epoch) AttributeError: 'list' object has no attribute 'add_scalar' ''' as i dig back, the RansacPnP output is CV ERROR, the point 2d and point 3d of input is empty in comute_query_pose_errors(data, preds) in eval_utils.py

the mscores0 and mscores1 are all close to zero which cannot pass the match_threshold
where max0 is tensor([[2.8489e-05, 1.0589e-04, 8.0255e-05, 1.7513e-05, 9.2613e-06, 1.4361e-04, 1.9696e-05, 2.7727e-05, 1.8381e-05, 1.2542e-05, 3.1094e-05, 9.6980e-06, 1.3867e-05, 1.0959e-05, 4.4529e-05, 1.8459e-05, 4.1601e-05, 6.6580e-05, 8.4602e-05, ..., 6.7834e-05, 4.2399e-05, 3.7876e-05, 2.7925e-05]], device='cuda:0') max1 is tensor([[156, 170, 64, ..., 286, 286, 286]], device='cuda:0')

Is this a problem with the colmap or seems like the problem with the superglue matcher?

DeriZSY commented 1 year ago

Hi, thanks for your interest in our work. Have you visualized the 2D keypoints and matched pairs of SuperGlue? From my experience, this may mostly be due to wrong keypoints and matches. Please check if the pre-train weights for SuperPoints and SuperGlue are on the right places and have successfully been loaded.

SylarZiling commented 1 year ago

Hi, thanks for your interest in our work. Have you visualized the 2D keypoints and matched pairs of SuperGlue? From my experience, this may mostly be due to wrong keypoints and matches. Please check if the pre-train weights for SuperPoints and SuperGlue are on the right places and have successfully been loaded.

Thank you for the reply. I double checked the superpoints and superglue, they are in the right place and have been successfully been load. The problem is the training process cannot pass the validation sanity check in pytorch lightining. it could also be the problem with colmap. I haven't visualize the 2d keypoints and matched pairs of superglue. How can I visualize that? through colmap? also i think the problem is in the validation sanity check, the 2d query des cannot match with the 3d des.

DeriZSY commented 1 year ago

Hi, thanks for your interest in our work. Have you visualized the 2D keypoints and matched pairs of SuperGlue? From my experience, this may mostly be due to wrong keypoints and matches. Please check if the pre-train weights for SuperPoints and SuperGlue are on the right places and have successfully been loaded.

Thank you for the reply. I double checked the superpoints and superglue, they are in the right place and have been successfully been load. The problem is the training process cannot pass the validation sanity check in pytorch lightining. it could also be the problem with colmap. I haven't visualize the 2d keypoints and matched pairs of superglue. How can I visualize that? through colmap? also i think the problem is in the validation sanity check, the 2d query des cannot match with the 3d des.

"RansacPnP output is CV ERROR" usually means a failure of solving PnP due to wrong 2D-3D matches where a correct pose cannot be obtained. You can try to visualize the matches with our tool: https://github.com/zju3dv/Wis3D Visualizing keypints and matches will usually boost the efficiency of debugging.

SylarZiling commented 1 year ago

Hi, thanks for your interest in our work. Have you visualized the 2D keypoints and matched pairs of SuperGlue? From my experience, this may mostly be due to wrong keypoints and matches. Please check if the pre-train weights for SuperPoints and SuperGlue are on the right places and have successfully been loaded.

Thank you for the reply. I double checked the superpoints and superglue, they are in the right place and have been successfully been load. The problem is the training process cannot pass the validation sanity check in pytorch lightining. it could also be the problem with colmap. I haven't visualize the 2d keypoints and matched pairs of superglue. How can I visualize that? through colmap? also i think the problem is in the validation sanity check, the 2d query des cannot match with the 3d des.

"RansacPnP output is CV ERROR" usually means a failure of solving PnP due to wrong 2D-3D matches where a correct pose cannot be obtained. You can try to visualize the matches with our tool: https://github.com/zju3dv/Wis3D Visualizing keypints and matches will usually boost the efficiency of debugging.

Yes, I tracked back the CV ERROR from RansacPnP and I found out the input mkpts2d and mkpts3d are both empty [] because the preds['matches0'] are all -1, which indicate invalid match. where the the matching confidence C_{3D} is too low to pass the threshhold

My question is, this error happened between 2d db and 3d db match, it appears in the validation sanity check, so before the training process. I am using the datasets you guys provided in the cloud drive for training and validation. func validation_step(self, batch, bach_idx):

  -> preds, conf_matrix_pred = self.matcher(data): 
         -> GATs_SuperGlue.forward():
                 -> conf_matrix = F.softmax(scores, 1) * F.softmax(scores, 2) # result is too low 

  -> pose_pred,val_results = compute_query_pose_errors(data,preds) :
         -> matches0 = preds['matches0'].cpu().numpy() = [-1 -1 -1 -1 -1  , ..., -1 -1 -1 -1]
         -> mkpts2d = query_kpts2d[valid] = []
         -> mkpts3d = query_kpts3d[matches0[valid]]=[]

Thank you very much

Sorry i realized my previous misunderstanding. I double checked the 2d keypoints with the reporjection 2d points from 3d key points. I think from the digram i can tell many points are matched, but the conf_matrix tells differently: conf matirx is tensor([[[2.8148e-08, 4.9996e-07, 5.8917e-08, ..., 4.6866e-07, 4.3113e-06, 1.7342e-07], [3.6634e-08, 1.1595e-07, 5.8826e-07, ..., 6.2275e-08, 4.0601e-07, 4.0078e-07], [2.3035e-09, 1.6528e-08, 9.7449e-09, ..., 6.3665e-08, 1.3049e-07, 1.1748e-08], ..., [1.9265e-07, 2.1242e-07, 3.2788e-08, ..., 2.8662e-08, 1.9842e-07, 7.6186e-08], [3.6983e-06, 1.7338e-06, 2.3379e-08, ..., 5.9582e-08, 2.9911e-08, 1.4061e-07], [3.2538e-06, 7.5517e-07, 1.1411e-07, ..., 5.6047e-08, 1.1298e-07, 8.4030e-07]]], device='cuda:0') Black is the 2d keypoints from extractor and white is the reporjected 3d keypoints Screenshot from 2022-11-09 11-11-16

SylarZiling commented 1 year ago

Hi, thanks for your interest in our work. Have you visualized the 2D keypoints and matched pairs of SuperGlue? From my experience, this may mostly be due to wrong keypoints and matches. Please check if the pre-train weights for SuperPoints and SuperGlue are on the right places and have successfully been loaded.

Thank you for the reply. I double checked the superpoints and superglue, they are in the right place and have been successfully been load. The problem is the training process cannot pass the validation sanity check in pytorch lightining. it could also be the problem with colmap. I haven't visualize the 2d keypoints and matched pairs of superglue. How can I visualize that? through colmap? also i think the problem is in the validation sanity check, the 2d query des cannot match with the 3d des.

"RansacPnP output is CV ERROR" usually means a failure of solving PnP due to wrong 2D-3D matches where a correct pose cannot be obtained. You can try to visualize the matches with our tool: https://github.com/zju3dv/Wis3D Visualizing keypints and matches will usually boost the efficiency of debugging.

Sorry I would also like to ask what is the scale_factor appeared in the train_GATsSPG.yaml file model section? I noticed this scale factor was used to divide the score after the dual-softmax. why it is 0.07?

SylarZiling commented 1 year ago

Hi, thanks for your interest in our work. Have you visualized the 2D keypoints and matched pairs of SuperGlue? From my experience, this may mostly be due to wrong keypoints and matches. Please check if the pre-train weights for SuperPoints and SuperGlue are on the right places and have successfully been loaded.

Thank you for the reply. I double checked the superpoints and superglue, they are in the right place and have been successfully been load. The problem is the training process cannot pass the validation sanity check in pytorch lightining. it could also be the problem with colmap. I haven't visualize the 2d keypoints and matched pairs of superglue. How can I visualize that? through colmap? also i think the problem is in the validation sanity check, the 2d query des cannot match with the 3d des.

"RansacPnP output is CV ERROR" usually means a failure of solving PnP due to wrong 2D-3D matches where a correct pose cannot be obtained. You can try to visualize the matches with our tool: https://github.com/zju3dv/Wis3D Visualizing keypints and matches will usually boost the efficiency of debugging.

Sorry I double check the problem and although i decrease the scale factor a bit and i have a valid conf_matirx, I am still running into the problem with the valid epoch end

File "OnePose01/src/models/GATsSPG_lightning_model.py", line 113, in validation_epoch_end self.logger.experiment.addscalar(f'metrics{valset_idx}/{k}', v, global_step=cur_epoch) AttributeError: 'list' object has no attribute 'add_scalar'

I double checked with pytorch lightning and pytorch, this experience is a torch.utils.tensorboard.SummaryWriter object.

I am wondering whether you guys faced similar situation before?

AnukritiSinghh commented 1 year ago

Hi @SylarZiling, can you please tell how you projected the 2d and 3d points into the image? I mean, can you share how you got those images exactly? I need that kind of visualization as well

zxl55552 commented 1 year ago

hello ，Is the problem solved?

zxl55552 commented 1 year ago

Hi, thanks for your interest in our work. Have you visualized the 2D keypoints and matched pairs of SuperGlue? From my experience, this may mostly be due to wrong keypoints and matches. Please check if the pre-train weights for SuperPoints and SuperGlue are on the right places and have successfully been loaded.

Thank you for the reply. I double checked the superpoints and superglue, they are in the right place and have been successfully been load. The problem is the training process cannot pass the validation sanity check in pytorch lightining. it could also be the problem with colmap. I haven't visualize the 2d keypoints and matched pairs of superglue. How can I visualize that? through colmap? also i think the problem is in the validation sanity check, the 2d query des cannot match with the 3d des.

"RansacPnP output is CV ERROR" usually means a failure of solving PnP due to wrong 2D-3D matches where a correct pose cannot be obtained. You can try to visualize the matches with our tool: https://github.com/zju3dv/Wis3D Visualizing keypints and matches will usually boost the efficiency of debugging.

Sorry I double check the problem and although i decrease the scale factor a bit and i have a valid conf_matirx, I am still running into the problem with the valid epoch end

File "OnePose01/src/models/GATsSPG_lightning_model.py", line 113, in validation_epoch_end self.logger.experiment.addscalar(f'metrics{valset_idx}/{k}', v, global_step=cur_epoch) AttributeError: 'list' object has no attribute 'add_scalar'

I double checked with pytorch lightning and pytorch, this experience is a torch.utils.tensorboard.SummaryWriter object.

I am wondering whether you guys faced similar situation before? hello ，Is the problem solved?

Jingranxia commented 1 year ago

Hi I am running into some problem when i am trying the training process.嗨，我在尝试训练过程时遇到了一些问题。 my env: ubuntu 20.04 & 18.04我的环境：Ubuntu 20.04 和 18.04 I tried both environment and both has the same error at validation sanity check process我尝试了两种环境，并且在验证健全性检查过程中都有相同的错误 the direct output is直接输出为 ''' ''' in validation_epoch_end 在validation_epoch_end self.logger.experiment.addscalar(f'metrics{valset_idx}/{k}', v, global_step=cur_epoch)self.logger.experiment.addscalar（f'metrics{valset_idx}/{k}'， v， global_step=cur_epoch） AttributeError: 'list' object has no attribute 'add_scalar'属性错误：“列表”对象没有属性“add_scalar” ''' ''' as i dig back, the RansacPnP output is CV ERROR, the point 2d and point 3d of input is empty in comute_query_pose_errors(data, preds) in eval_utils.py当我回头挖掘时，RansacPnP 输出是 CV 错误，输入的点 2d 和点 3d 在comute_query_pose_errors（数据、preds）中为空eval_utils.py

the mscores0 and mscores1 are all close to zero which cannot pass the match_thresholdmscores0 和 mscores1 都接近于零，无法通过match_threshold where 哪里 max0 is tensor([[2.8489e-05, 1.0589e-04, 8.0255e-05, 1.7513e-05, 9.2613e-06, 1.4361e-04,max0 是张量（[[2.8489e-05， 1.0589e-04， 8.0255e-05， 1.7513e-05， 9.2613e-06， 1.4361e-04， 1.9696e-05, 2.7727e-05, 1.8381e-05, 1.2542e-05, 3.1094e-05, 9.6980e-06,1.9696e-05， 2.7727e-05， 1.8381e-05， 1.2542e-05， 3.1094e-05， 9.6980e-06， 1.3867e-05, 1.0959e-05, 4.4529e-05, 1.8459e-05, 4.1601e-05, 6.6580e-05,1.3867e-05， 1.0959e-05， 4.4529e-05， 1.8459e-05， 4.1601e-05， 6.6580e-05， 8.4602e-05, ..., 6.7834e-05, 4.2399e-05, 3.7876e-05,8.4602E-05， ...， 6.7834E-05， 4.2399E-05， 3.7876E-05， 2.7925e-05]], device='cuda:0')2.7925e-05]]， device='cuda：0'） max1 is tensor([[156, 170, 64, ..., 286, 286, 286]], device='cuda:0')max1 is tensor（[[156， 170， 64， ...， 286， 286， 286]]， device='cuda：0'）

Is this a problem with the colmap or seems like the problem with the superglue matcher?这是colmap的问题还是强力胶匹配器的问题？

Hi I am running into some problem when i am trying the training process. my env: ubuntu 20.04 & 18.04 I tried both environment and both has the same error at validation sanity check process the direct output is ''' in validation_epoch_end self.logger.experiment.addscalar(f'metrics{valset_idx}/{k}', v, global_step=cur_epoch) AttributeError: 'list' object has no attribute 'add_scalar' ''' as i dig back, the RansacPnP output is CV ERROR, the point 2d and point 3d of input is empty in comute_query_pose_errors(data, preds) in eval_utils.py

the mscores0 and mscores1 are all close to zero which cannot pass the match_threshold where max0 is tensor([[2.8489e-05, 1.0589e-04, 8.0255e-05, 1.7513e-05, 9.2613e-06, 1.4361e-04, 1.9696e-05, 2.7727e-05, 1.8381e-05, 1.2542e-05, 3.1094e-05, 9.6980e-06, 1.3867e-05, 1.0959e-05, 4.4529e-05, 1.8459e-05, 4.1601e-05, 6.6580e-05, 8.4602e-05, ..., 6.7834e-05, 4.2399e-05, 3.7876e-05, 2.7925e-05]], device='cuda:0') max1 is tensor([[156, 170, 64, ..., 286, 286, 286]], device='cuda:0')

Is this a problem with the colmap or seems like the problem with the superglue matcher?

Hello, I have encountered the same problem as you. May I ask how you resolved it later? This is very important to me and I hope to receive a response. After I solve the CVERROR problem,The problem with add_ scaler has always existed, and I don't know where the problem is. If possible, could you please add your WeChat communication to it。

zxl55552 commented 1 year ago

我也还没有解决那个问题，不过那个属性错误我看是写日志的，那部分我给注释掉了，然后代码能跑通

Hi I am running into some problem when i am trying the training process.嗨，我在尝试训练过程时遇到了一些问题。 my env: ubuntu 20.04 & 18.04我的环境：Ubuntu 20.04 和 18.04 I tried both environment and both has the same error at validation sanity check process我尝试了两种环境，并且在验证健全性检查过程中都有相同的错误 the direct output is直接输出为 ''' ''' in validation_epoch_end 在validation_epoch_end self.logger.experiment.addscalar(f'metrics{valset_idx}/{k}', v, global_step=cur_epoch)self.logger.experiment.addscalar（f'metrics{valset_idx}/{k}'， v， global_step=cur_epoch） AttributeError: 'list' object has no attribute 'add_scalar'属性错误：“列表”对象没有属性“add_scalar” ''' ''' as i dig back, the RansacPnP output is CV ERROR, the point 2d and point 3d of input is empty in comute_query_pose_errors(data, preds) in eval_utils.py当我回头挖掘时，RansacPnP 输出是 CV 错误，输入的点 2d 和点 3d 在comute_query_pose_errors（数据、preds）中为空eval_utils.py

the mscores0 and mscores1 are all close to zero which cannot pass the match_thresholdmscores0 和 mscores1 都接近于零，无法通过match_threshold where 哪里 max0 is tensor([[2.8489e-05, 1.0589e-04, 8.0255e-05, 1.7513e-05, 9.2613e-06, 1.4361e-04,max0 是张量（[[2.8489e-05， 1.0589e-04， 8.0255e-05， 1.7513e-05， 9.2613e-06， 1.4361e-04， 1.9696e-05, 2.7727e-05, 1.8381e-05, 1.2542e-05, 3.1094e-05, 9.6980e-06,1.9696e-05， 2.7727e-05， 1.8381e-05， 1.2542e-05， 3.1094e-05， 9.6980e-06， 1.3867e-05, 1.0959e-05, 4.4529e-05, 1.8459e-05, 4.1601e-05, 6.6580e-05,1.3867e-05， 1.0959e-05， 4.4529e-05， 1.8459e-05， 4.1601e-05， 6.6580e-05， 8.4602e-05, ..., 6.7834e-05, 4.2399e-05, 3.7876e-05,8.4602E-05， ...， 6.7834E-05， 4.2399E-05， 3.7876E-05， 2.7925e-05]], device='cuda:0')2.7925e-05]]， device='cuda：0'） max1 is tensor([[156, 170, 64, ..., 286, 286, 286]], device='cuda:0')max1 is tensor（[[156， 170， 64， ...， 286， 286， 286]]， device='cuda：0'）

Is this a problem with the colmap or seems like the problem with the superglue matcher?这是colmap的问题还是强力胶匹配器的问题？

Hi I am running into some problem when i am trying the training process. my env: ubuntu 20.04 & 18.04 I tried both environment and both has the same error at validation sanity check process the direct output is ''' in validation_epoch_end self.logger.experiment.addscalar(f'metrics{valset_idx}/{k}', v, global_step=cur_epoch) AttributeError: 'list' object has no attribute 'add_scalar' ''' as i dig back, the RansacPnP output is CV ERROR, the point 2d and point 3d of input is empty in comute_query_pose_errors(data, preds) in eval_utils.py

the mscores0 and mscores1 are all close to zero which cannot pass the match_threshold where max0 is tensor([[2.8489e-05, 1.0589e-04, 8.0255e-05, 1.7513e-05, 9.2613e-06, 1.4361e-04, 1.9696e-05, 2.7727e-05, 1.8381e-05, 1.2542e-05, 3.1094e-05, 9.6980e-06, 1.3867e-05, 1.0959e-05, 4.4529e-05, 1.8459e-05, 4.1601e-05, 6.6580e-05, 8.4602e-05, ..., 6.7834e-05, 4.2399e-05, 3.7876e-05, 2.7925e-05]], device='cuda:0') max1 is tensor([[156, 170, 64, ..., 286, 286, 286]], device='cuda:0')

Is this a problem with the colmap or seems like the problem with the superglue matcher?

Hello, I have encountered the same problem as you. May I ask how you resolved it later? This is very important to me and I hope to receive a response. After I solve the CVERROR problem,The problem with add_ scaler has always existed, and I don't know where the problem is. If possible, could you please add your WeChat communication to it。

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

Jingranxia commented 1 year ago

能跑通了，训练出来的结果如何

夏靖然 @.***

------------------ 原始邮件 ------------------ 发件人: "zju3dv/OnePose" @.>; 发送时间: 2023年9月14日(星期四) 下午3:07 @.>; @.**@.>; 主题: Re: [zju3dv/OnePose] Error in training process with validation check (Issue #25)

我也还没有解决那个问题，不过那个属性错误我看是写日志的，那部分我给注释掉了，然后代码能跑通

Hi I am running into some problem when i am trying the training process.嗨，我在尝试训练过程时遇到了一些问题。 my env: ubuntu 20.04 & 18.04我的环境：Ubuntu 20.04 和 18.04 I tried both environment and both has the same error at validation sanity check process我尝试了两种环境，并且在验证健全性检查过程中都有相同的错误 the direct output is直接输出为 ''' ''' in validation_epoch_end 在validation_epoch_end self.logger.experiment.addscalar(f'metrics{valset_idx}/{k}', v, global_step=cur_epoch)self.logger.experiment.addscalar（f'metrics{valset_idx}/{k}'， v， global_step=cur_epoch） AttributeError: 'list' object has no attribute 'add_scalar'属性错误：“列表”对象没有属性“add_scalar” ''' ''' as i dig back, the RansacPnP output is CV ERROR, the point 2d and point 3d of input is empty in comute_query_pose_errors(data, preds) in eval_utils.py当我回头挖掘时，RansacPnP 输出是 CV 错误，输入的点 2d 和点 3d 在comute_query_pose_errors（数据、preds）中为空eval_utils.py

the mscores0 and mscores1 are all close to zero which cannot pass the match_thresholdmscores0 和 mscores1 都接近于零，无法通过match_threshold where 哪里 max0 is tensor([[2.8489e-05, 1.0589e-04, 8.0255e-05, 1.7513e-05, 9.2613e-06, 1.4361e-04,max0 是张量（[[2.8489e-05， 1.0589e-04， 8.0255e-05， 1.7513e-05， 9.2613e-06， 1.4361e-04， 1.9696e-05, 2.7727e-05, 1.8381e-05, 1.2542e-05, 3.1094e-05, 9.6980e-06,1.9696e-05， 2.7727e-05， 1.8381e-05， 1.2542e-05， 3.1094e-05， 9.6980e-06， 1.3867e-05, 1.0959e-05, 4.4529e-05, 1.8459e-05, 4.1601e-05, 6.6580e-05,1.3867e-05， 1.0959e-05， 4.4529e-05， 1.8459e-05， 4.1601e-05， 6.6580e-05， 8.4602e-05, ..., 6.7834e-05, 4.2399e-05, 3.7876e-05,8.4602E-05， ...， 6.7834E-05， 4.2399E-05， 3.7876E-05， 2.7925e-05]], device='cuda:0')2.7925e-05]]， device='cuda：0'） max1 is tensor([[156, 170, 64, ..., 286, 286, 286]], device='cuda:0')max1 is tensor（[[156， 170， 64， ...， 286， 286， 286]]， device='cuda：0'）

Is this a problem with the colmap or seems like the problem with the superglue matcher?这是colmap的问题还是强力胶匹配器的问题？

Hi I am running into some problem when i am trying the training process. my env: ubuntu 20.04 & 18.04 I tried both environment and both has the same error at validation sanity check process the direct output is ''' in validation_epoch_end self.logger.experiment.addscalar(f'metrics{valset_idx}/{k}', v, global_step=cur_epoch) AttributeError: 'list' object has no attribute 'add_scalar' ''' as i dig back, the RansacPnP output is CV ERROR, the point 2d and point 3d of input is empty in comute_query_pose_errors(data, preds) in eval_utils.py

the mscores0 and mscores1 are all close to zero which cannot pass the match_threshold where max0 is tensor([[2.8489e-05, 1.0589e-04, 8.0255e-05, 1.7513e-05, 9.2613e-06, 1.4361e-04, 1.9696e-05, 2.7727e-05, 1.8381e-05, 1.2542e-05, 3.1094e-05, 9.6980e-06, 1.3867e-05, 1.0959e-05, 4.4529e-05, 1.8459e-05, 4.1601e-05, 6.6580e-05, 8.4602e-05, ..., 6.7834e-05, 4.2399e-05, 3.7876e-05, 2.7925e-05]], device='cuda:0') max1 is tensor([[156, 170, 64, ..., 286, 286, 286]], device='cuda:0')

Is this a problem with the colmap or seems like the problem with the superglue matcher?

Hello, I have encountered the same problem as you. May I ask how you resolved it later? This is very important to me and I hope to receive a response. After I solve the CVERROR problem,The problem with add_ scaler has always existed, and I don't know where the problem is. If possible, could you please add your WeChat communication to it。

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.> — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

zxl55552 commented 1 year ago

论文中的这些数据集区分大物体，中物体和小物体，但是代码中并没有区分，不过我看整体的跟论文中差距不大