search

wenbowen123 / iros20-6d-pose-tracking

[IROS 2020] se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains
Other
384 stars 66 forks source link

Array shape mismatch when rendering #53

Closed marcusnagy closed 1 year ago

marcusnagy commented 1 year ago 
sudo ./predict.sh 
ckpt_dir: /home/bowen/debug/bleach_cleanser/model_best_val.pth.tar
dataset_info_path /media/bowen/e25c9489-2f57-42dd-b076-021c59369fec/DATASET/train_example/bleach_cleanser/train_data_blender_DR/../dataset_info.yml
/media/bowen/e25c9489-2f57-42dd-b076-021c59369fec/DATASET/YCBInEOAT/bleach0
self.object_cloud loaded and downsampled
self.object_width= 285.37860394994397
Loading ckpt from  /home/bowen/debug/bleach_cleanser/model_best_val.pth.tar
pose track ckpt epoch=112
Using vispy renderer
model_path:  /media/bowen/e25c9489-2f57-42dd-b076-021c59369fec/DATASET/YCB_Video_Dataset/CADmodels/021_bleach_cleanser/textured.ply
self.cam_K:
 [[1.066778e+03 0.000000e+00 3.129869e+02]
 [0.000000e+00 1.067487e+03 2.413109e+02]
 [0.000000e+00 0.000000e+00 1.000000e+00]]
making dataset... for eval
#dataset: 0
self.trans_normalizer=0.03, self.rot_normalizer=0.5235987755982988
init pose
 [[-0.6981918   0.05113025  0.71419933 -0.39318141]
 [ 0.57201178  0.6396664   0.51329966  0.27422776]
 [-0.43062506  0.76665225 -0.47589299  0.83633948]
 [ 0.          0.          0.          1.        ]]
Offset Left: 371
Offset Right: 364
Left: 0
 Right: -7
Traceback (most recent call last):
  File "/home/se3_tracknet/predict.py", line 641, in <module>
    predictSequenceMyData()
  File "/home/se3_tracknet/predict.py", line 592, in predictSequenceMyData
    cur_pose = tracker.on_track(A_in_cam, rgb, depth, gt_A_in_cam=np.eye(4),gt_B_in_cam=np.eye(4), debug=debug,samples=samples)
  File "/home/se3_tracknet/predict.py", line 217, in on_track
    rgbB, depthB = crop_bbox(current_rgb, current_depth, bb, self.image_size)
  File "/home/se3_tracknet/Utils.py", line 341, in crop_bbox
    color_crop[top_offset:bottom_offset, left_offset:right_offset, :] = color[top:bottom, left:right, :]
ValueError: could not broadcast input array from shape (71,633,3) into shape (71,0,3)
martinlyra commented 1 year ago

When we looked closer at this to debug the issue, this error seemed to happen when the bounding box had negative Y on all 4 points. Does it mean the bounding box was outside the picture?

wenbowen123 commented 1 year ago

Hi, did you try other objects than bleach_cleanser and does it happen as well?

marcusnagy commented 1 year ago

Hey! I tried with the power_drill and get the same result, added a few prints that could be helpful.

sudo ./predict.sh 
ckpt_dir: /home/marcusmartin/debug/power_drill/model_epoch215.pth.tar
dataset_info_path /media/marcusmartin/e25c9489-2f57-42dd-b076-021c59369fec/DATASET/train_example/power_drill/train_data_blender_DR/../dataset_info.yml
/media/marcusmartin/e25c9489-2f57-42dd-b076-021c59369fec/DATASET/YCBInEOAT/bleach0
self.object_cloud loaded and downsampled
self.object_width= 248.7782030176937
Loading ckpt from  /home/marcusmartin/debug/power_drill/model_epoch215.pth.tar
pose track ckpt epoch=215
Using vispy renderer
model_path:  /media/marcusmartin/e25c9489-2f57-42dd-b076-021c59369fec/DATASET/YCB_Video_Dataset/CADmodels/035_power_drill/textured.ply
self.cam_K:
 [[1.066778e+03 0.000000e+00 3.129869e+02]
 [0.000000e+00 1.067487e+03 2.413109e+02]
 [0.000000e+00 0.000000e+00 1.000000e+00]]
making dataset... for eval
#dataset: 0
self.trans_normalizer=0.03, self.rot_normalizer=0.5235987755982988
init pose
 [[-0.6981918   0.05113025  0.71419933 -0.39318141]
 [ 0.57201178  0.6396664   0.51329966  0.27422776]
 [-0.43062506  0.76665225 -0.47589299  0.83633948]
 [ 0.          0.          0.          1.        ]]
THIS IS BOUDING BOX: [[ 433 -347]
 [ 750 -347]
 [ 433  -30]
 [ 750  -30]]
Offset Left: 347
Offset Right: 317
Left: 0
 Right: -30
Traceback (most recent call last):
  File "/home/marcusmartin/repos/iros20-6d-pose-tracking/predict.py", line 649, in <module>
    predictSequenceMyData()
  File "/home/marcusmartin/repos/iros20-6d-pose-tracking/predict.py", line 592, in predictSequenceMyData
    cur_pose = tracker.on_track(A_in_cam, rgb, depth, gt_A_in_cam=np.eye(4),gt_B_in_cam=np.eye(4), debug=debug,samples=samples)
  File "/home/marcusmartin/repos/iros20-6d-pose-tracking/predict.py", line 217, in on_track
    rgbB, depthB = crop_bbox(current_rgb, current_depth, bb, self.image_size)
  File "/home/marcusmartin/repos/iros20-6d-pose-tracking/Utils.py", line 342, in crop_bbox
    color_crop[top_offset:bottom_offset, left_offset:right_offset, :] = color[top:bottom, left:right, :]
ValueError: could not broadcast input array from shape (47,610,3) into shape (47,0,3)
martinlyra commented 1 year ago

This problem seem to do only appear when using the predictSequenceMyData() in predict.py, when we were using predictSequenceYcb() it seems to work fine.

wenbowen123 commented 1 year ago

predictSequenceMyData is used for running on YCBInEOAT dataset predictSequenceYcb is used for running on YCB-Video dataset. They are also using different ckpts for the same objects. So make sure when you change the dataset, you are using the right functions and ckpts. https://github.com/wenbowen123/iros20-6d-pose-tracking/tree/master#data-download

wenbowen123 commented 1 year ago

@martinlyra @marcusnagy We recently made a fix. Please follow the new readme.