Plot semantic segmentation multi-class masks on RGB image

[asmagen]

I tried using the Visualizer module to plot my predicted segmentation masks (binary channels) on an RGB image, but it seems to require some kind of detectron2 specific representation.

What's the easiest way to plot the segmentation mask overlays if I don't use the detectron2 pipeline for the actual segmentation?

[ppwwyyxx]

It's documented at https://detectron2.readthedocs.io/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_sem_seg

[asmagen]

I tried using this way but it doesn't seem to work without the metadata object. How do I use it without any metadata and just assign random colors to different classes?

Thanks

[ppwwyyxx]

Sounds like a reasonable feature to add to allow draw_sem_seg to work without metadata.

Currently you can use lower level drawing functions, such as draw_binary_mask.

[asmagen]

Thanks but is there a way to create a mock or empty metadata object to make it work in the meanwhile?

[ppwwyyxx]

Use draw_binary_mask:

for x in np.unique(sem_seg):
    visualizer.draw_binary_mask(sem_seg == x)

[asmagen]

Any idea why I'm getting the following error?

Thanks

[ppwwyyxx]

The correct APIs are documented in https://detectron2.readthedocs.io/modules/utils.html.

[Ellyuca]

just my 2-cents, maybe it can be helpful for someone in the future. Following the suggestion of @ppwwyyxx , to get the mask from the semantic segmentation I've done something like this:

cfg = get_cfg()
add_deeplab_config(cfg)
add_maskformer2_config(cfg)
cfg.merge_from_file("configs/coco/panoptic-segmentation/swin/maskformer2_swin_large_IN21k_384_bs16_100ep.yaml")
cfg.MODEL.WEIGHTS = 'https://dl.fbaipublicfiles.com/maskformer/mask2former/coco/panoptic/maskformer2_swin_large_IN21k_384_bs16_100ep/model_final_f07440.pkl'
cfg.MODEL.MASK_FORMER.TEST.SEMANTIC_ON = True
cfg.MODEL.MASK_FORMER.TEST.INSTANCE_ON = True
cfg.MODEL.MASK_FORMER.TEST.PANOPTIC_ON = True
predictor = DefaultPredictor(cfg)

im = cv2.imread("path_to_image")
outputs = predictor(im) 
sem_seg = outputs["sem_seg"].argmax(0).to("cpu")

for x in np.unique(sem_seg):
    temp = np.zeros_like(im)
    v = Visualizer(temp, coco_metadata, scale=1.2, instance_mode=ColorMode.IMAGE_BW)
    mask = v.draw_binary_mask(np.array(sem_seg) == x )
    cv2_imshow(mask.get_image())