Array size mismatch when running `predict_ros.py`

[martinlyra]

1684482199196819782: RGB: 480 - D: 480
ERROR: Traceback (most recent call last):
  File "predict_ros.py", line 117, in <module>
    ros_tracker.on_track()
  File "predict_ros.py", line 59, in on_track
    ob_in_cam = self.tracker.on_track(self.A_in_cam,self.color.astype(np.uint8), self.depth, gt_A_in_cam=np.eye(4),gt_B_in_cam=np.eye(4), debug=False,samples=1)
  File "/home/marcusmartin/iros20-tracking/predict.py", line 217, in on_track
    rgbB, depthB = crop_bbox(current_rgb, current_depth, bb, self.image_size)
  File "/home/marcusmartin/iros20-tracking/Utils.py", line 405, in crop_bbox
    depth_crop[top_offset:bottom_offset, left_offset:right_offset] = depth[top:bottom, left:right]
ValueError: could not broadcast input array from shape (0,0,3) into shape (0,0)

We run with slight modification to the logging to allow us see what is running under the hood. Like this one at start of on_track:

print(f"{self.cur_time}: RGB: {len(self.color) if not self.color is None else 0} - D: {len(self.depth) if not self.depth is None else 0}")

This is happening on our own training data as mentioned in #55. Is this an issue with our data? Or is it an issue caused by Open3D, as we get an warning regarding the model not being read in due to unknown file format.

Or is it an issue with our Realsense camera? The model we are using is a D435. Here is an echo of from our realsense/aligned_depth_to_color/camera_info if it helps

header: 
  seq: 24338
  stamp: 
    secs: 1684481948
    nsecs: 714720011
  frame_id: "realsense_rgb_optical_frame"
height: 480
width: 848
distortion_model: "plumb_bob"
D: [0.0, 0.0, 0.0, 0.0, 0.0]
K: [614.9791259765625, 0.0, 430.603271484375, 0.0, 615.01416015625, 237.27053833007812, 0.0, 0.0, 1.0]
R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
P: [614.9791259765625, 0.0, 430.603271484375, 0.0, 0.0, 615.01416015625, 237.27053833007812, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---

The encoding of the messages from the aligned-depth-to-color topic is 16UC1, same for the depth.

[martinlyra]

Update, we figured out the array-mismatch issue: https://github.com/wenbowen123/iros20-6d-pose-tracking/blob/d645214c1a06554d1389f202e842745abaa865eb/predict_ros.py#L26-L27 The topics seem to be flipped around. (depth image topic for colour, and colour image topic for depth), putting them in the right places seem to fix it.

However, one part of the original issue where the corp produces a completely empty image, 0 by 0 pixels, seems to remain?

ERROR: Traceback (most recent call last):
  File "predict_ros.py", line 124, in <module>
    ros_tracker.on_track()
  File "predict_ros.py", line 66, in on_track
    ob_in_cam = self.tracker.on_track(self.A_in_cam,self.color.astype(np.uint8), self.depth, gt_A_in_cam=np.eye(4),gt_B_in_cam=np.eye(4), debug=False,samples=1)
  File "/home/marcusmartin/iros20-tracking/predict.py", line 219, in on_track
    rgbB, depthB = crop_bbox(current_rgb, current_depth, bb, self.image_size)
  File "/home/marcusmartin/iros20-tracking/Utils.py", line 406, in crop_bbox
    resized_rgb = cv2.resize(color_crop, output_size, interpolation=cv2.INTER_NEAREST)
cv2.error: OpenCV(4.7.0) /io/opencv/modules/imgproc/src/resize.cpp:4062: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

[martinlyra]

I do notice that the pose_init we have is an identity matrix without rotation (3 by 3 identity) and no translation. I assume we need to get an initial pose by estimating the pose by different means?

[wenbowen123]

Yes, the pose initialization is not included by se(3)-tracknet, which only tracks the pose afterwards. To initialize, you need other methods. For instance, if you want to do a quick testing over the video offline, you can try annotating the initial pose using tools such as https://github.com/sakizuki/6DPoseAnnotator