If I want to merge pspnet into the faster rcnn,what should I do?

[gittigxuy]

I want to use faster rcnn to do detection,and use pspnet to do segment,how could I merge these two model into one model?

[fmassa]

Hi,

I suppose you want to do something like panoptic segmentation, is that right? There are multiple ways of approaching this problem, and this is a research question.

I first implementation idea would be to create a new folder segmentation_head in the roi_heads folder, and implement the necessary functions. Note that each ROIHead takes as arguments a set of feature maps and proposals. You might want to use the proposals or not to help your model.

The other point you might need to change is that you now will want to return boxes and a full segmentation map, so this doesn't fit into the GeneralizedRCNN framework.

If you have a better idea of a paper you want to implement, I can give more precise help.

[gittigxuy]

Thanks ,how about this issue in detectron?https://github.com/facebookresearch/Detectron/issues/673,same ways to solve?

[fmassa]

From a quick look, it seems that the paper referenced in that issue, "Panoptic Segmentation", only provides a baseline implementation where there is no joint training of the segmentation masks and the stuff class. If you want to replicate their results, you could take the output of the models provided here and combine with the results from a semantic segmentation model, and apply the algorithm that they propose in section 7.

[fmassa]

For reference, the paper is in https://arxiv.org/abs/1801.00868

So to replicate their results, you don't need to modify the implementation in this repository

[kabjah]

how can i combine PSPNet with Mask-RCNN for image segmentation and detection