Multiple Removals at once

[hadillivvy]

Does this library have the option of removing more than one element from the scene at once? In configs/prepare_data/sam_text.json we can see that the text prompt is an array which seems to suggest that if we supply different text prompts here, all the objects in the scene matching the prompts should be deleted but that's not the case. Changing the code to accept multiple masks at once seems to break the refine step when running the gen_lama_prior.sh script.

Can anyone advise how to achieve multiple object removals in the same pipeline run?

[cuteyyt]

You may check configs/prepare_data/lama.json and modify the value of key 'num_masks' to match the number of your target objects. I am refining this project for the other questions mentioned in the issues.

[hadillivvy]

Hello @cuteyyt thanks for getting back to me! I did update the num_masks to the number of objects, I also updated the code in run_sam_text.py because in the original code when there's more than one text prompt, the masks produced were overriding each other and the output of the script was only the bounding box, image with mask and mask of the last prompt. So I updated the code to make it such that the script produces multiple bounding boxes, multiple masks in the images and multiple masks in the same mask. However, when running the script run_sam_points.py to map the masks to the other views, the resulting mapped masks were not clean and the so the assert(len(contours)) == num_masks was failing in the mask_refine function in mask_refine. I disabled the refining step but the final output ended up being a blur effect on the objects I want deleted and not a deletion of them. Any advice?

[cuteyyt]

If the prediction of SAM doesn't cover the target region you want to remove, you may dilate the mask. Or if the mask is not very "regular", with holes or jagged edges, but it is generally correct, then maybe try to utilize the SAM output logits and mask_input with multiple iterations. For masks passed to inpainting, it needs to be slightly bigger than the exact outline.

[hadillivvy]

Hi @cuteyyt Yes I'm having sometimes the issue of the mask having holes and this is resulting in a number of contours that's not equal to num_masks. Can you please explain more how I can do this? "utilize the SAM output logits and mask_input with multiple iterations"

[cuteyyt]

https://github.com/facebookresearch/segment-anything/issues/169 You may refer to the observations here, and in my practice, passing the return logits of SAM to its mask_input for several iterations may help. For example,

logits=None
for _ in range(iterations):
  masks, scores, logits = predictor.predict(
      ...
      mask_input=logits,
  )

[hadillivvy]

Hi @cuteyyt thank you! One more thing. Do you have an idea on how to handle the mapping of the mask in the gen_mask_sam_points script in the case where the object we want to delete is not in all of the scenes? Right now, the mask is being blown up to encompass the entire scene when in reality the issue is that in that particular shot, the object we want to remove is not present. Any idea where we can detect this in the code? I believe it should somewhere in mask_refine.py but not sure how to catch this case.

[cuteyyt]

I'm sorry, but this project currently does not consider this situation. If you want to use these views in subsequent steps, I think an easy way is to generate a whole black mask for the views without the target object. In this case, inpainting will not change the image.

[hadillivvy]

Hi @cuteyyt yes this is what i want to do but where in the code can we detect that the object is not in the image? Because I can see that in the images in imgs_with_points you managed to output images without no points if the object is not in the image and so there must be a way.

[cuteyyt]

For the text prompt, if there is no target object, the bounding box should be None or something like that. Do you mean the current code can run without pseudo-point annotation on a particular view for the points prompt? If the program cannot continue when no pseudo-point annotation is generated, you can manually insert code there to make a whole black mask. You may check the function pred_warp. The code in it like masks = predict_by_sam_single_img(predictor, img, points, labels, opt) calls SAM.

[hadillivvy]

Hi @cuteyyt OK got it to output a black mask. I just applied a filter on the points2d np array in run_sam_points.py and the filter checks if the points2d are within the x,y range of the image (i.e, each point's x has to be between 0 and the width of the image and each point's y has to be between 0 and height of the image). For images where the object is not present, we're getting a black mask and this is the desired outcome. The subsequent inpainting step worked well with the black mask.