facebookresearch / maskrcnn-benchmark

Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch.
MIT License
9.3k stars 2.49k forks source link

If I want to merge pspnet into the faster rcnn,what should I do? #39

Open gittigxuy opened 6 years ago

gittigxuy commented 6 years ago

I want to use faster rcnn to do detection,and use pspnet to do segment,how could I merge these two model into one model?

fmassa commented 6 years ago

Hi,

I suppose you want to do something like panoptic segmentation, is that right? There are multiple ways of approaching this problem, and this is a research question.

I first implementation idea would be to create a new folder segmentation_head in the roi_heads folder, and implement the necessary functions. Note that each ROIHead takes as arguments a set of feature maps and proposals. You might want to use the proposals or not to help your model.

The other point you might need to change is that you now will want to return boxes and a full segmentation map, so this doesn't fit into the GeneralizedRCNN framework.

If you have a better idea of a paper you want to implement, I can give more precise help.

gittigxuy commented 6 years ago

Thanks ,how about this issue in detectron?https://github.com/facebookresearch/Detectron/issues/673,same ways to solve?

fmassa commented 6 years ago

From a quick look, it seems that the paper referenced in that issue, "Panoptic Segmentation", only provides a baseline implementation where there is no joint training of the segmentation masks and the stuff class. If you want to replicate their results, you could take the output of the models provided here and combine with the results from a semantic segmentation model, and apply the algorithm that they propose in section 7.

fmassa commented 6 years ago

For reference, the paper is in https://arxiv.org/abs/1801.00868

So to replicate their results, you don't need to modify the implementation in this repository

kabjah commented 1 year ago

how can i combine PSPNet with Mask-RCNN for image segmentation and detection