Masks for object margin issue

[rohitdavas]

We can generate masks for objects and hands. ( This issue does not concern the hands masks as they use MANO and I can't use them due to license issue. )

When I am trying to generate masks for objects, there is margin of blue that is left. Can you give some suggestion what might be going wrong ?

Using grabcut to optimize gives result :

My logic :

      # create renderer
      object_renderer = rutils.DepthRenderer(object_name, cp.K(camera_name), camera_name, mesh_scale=1e-3)

      # render object
      object_pose = cp.object_pose(camera_name, frame_idx)
      object_rendering = object_renderer.render(object_pose)

      # create mask from rendering 
      object_mask = object_rendering > 0 

      # color img area of interest
      color_im_bgr_interest = color_im_bgr.copy()
      color_im_bgr_interest[np.logical_not(object_mask)] = 0 

      # due to hand object touch there are parts of hands that are included in 
      # mask.
      # since the objects are of blue color
      # based on mask refine the channel. 
      b, g, r = cv2.split(color_im_bgr_interest)
      color_mask = np.logical_and(b > g, b > r)

      # # dilate the color_mask to cover boundries better. 
      # color_mask = np.uint8(color_mask)
      # kernel = np.ones((5, 5), np.uint8)
      # color_mask = cv2.dilate(color_mask, kernel, iterations=1)

      # # convert to logical bool
      # color_mask = np.array(color_mask, dtype=np.bool)

      # object_mask = np.logical_and(object_mask, color_mask)
      object_mask = np.logical_and(object_mask, color_mask)

      object_mask = mutils.grabcut_mask(color_im_bgr, object_mask, n_iters=10

      mask = np.logical_not(object_mask)

[samarth-robo]

@rohitdavas if I understand correctly, you want that remaining thin blue region to also be included in the mask, right?

First, the general comment is that the mask is pretty good as it is :)

It can be improved. If you look at the grabcut_mask(rgb_im, mask) function, it initializes the GrabCut algorithm with

1. the pixels in mask as surely foreground
2. other pixels in the rectangular bounding box of the mask as probably background, and
3. the rest of the image as surely background

You can change (2) from probably background to probably foreground, or just remove that probably foreground/background initialization alltogether. See if that gives better results. If that does not work we can think of some other ideas.

[rohitdavas]

Thanks. I am closing issue for this. You understood it right. I opened the issue to understand what I am doing is good enough with what you have found while working on this.

Thanks for your comment.