Pytorch MaskRCNN

This is a Pytorch/Fastai implementation of MaskRCNN based on previous versions by Matterport and MultiModal Learning (see acknowledgements at bottom of page). This was mainly a personal learning exercise but the simplified structure may help others wanting to understand maskrcnn.

Includes:

• Overview diagram that shows the key components of maskRCNN
• Code cleaned up and structured as per the diagram to make it easy to understand and experiment with each part.
• Works with pytorch v1 and fastai v1
• Training and prediction working on nuke data from 2018 kaggle bowl; training and prediction on synthetic shapes data; and prediction on images using coco categories.

Todo:

• More tests. How can it best be tested?
• Could the experimental test framework be useful for continuous testing? Is there a good framework already written?
• How to expand testing to keras. i.e. how to adapt matterport for eager mode?

Structure

Note that training and prediction follow slightly different paths.

The core code reflects the diagram:

• datagen - anchors, dataset, head_targets, rpn_targets
• model - maskrcnn (whole thing), resnet (backbone), resnetFPN (feature pyramid), rpn
• filter - proposals, roialign, detections
• loss - loss functions for rpn and head

Utilities

• lib - c extensions for nms and roialign
• utils - box_utils, image_utils, visualize, batch (function decorator to process batches)
• callbacks - to tailor fastai for maskrcnn
• ipstartup - startup script for notebooks
• config - configuration constants

Samples

• Applications typically with dataset, config, learner, notebooks

Test (experimental)

• baseline - classes to help compare a new versus baseline version
• test_*.py
  • limited number of pytest functions versus multimodal
  • tried versus matterport but original needs adapting to work with eager mode
• tests.ipynb - notebook to run pytests

Installation

1. Clone this repository.

   git clone https://github.com/simonm3/maskr.git

2. Download pretrained coco weights from Google Drive.

3. Build the nms and roialign binaries:

   cd maskr/maskr/lib
   ./make.sh

4. Install the python package and dependencies in edit mode

   cd maskr
   pip install -e .

Acknowledgements and links

Blogs

These are useful introductions to image segmentation and maskrcnn

A Brief History of CNNs in Image Segmentation: From R-CNN to Mask R-CNN

Deep Learning for Instance-level Object Understanding. Tutorial on Deep Learning for Objects and Scenes (by Ross Girshick).

Object Detection using Deep Learning for advanced users

Faster R-CNN: Down the rabbit hole of modern object detection

Region of Interest Pooling explained

Papers

These are the main papers that led to maskrcnn:

FastRCNN

FasterRCNN

MaskRCNN

Feature Pyramid Networks

Packages

This package was based on the multimodal package which was based on matterport.

Matterport Mask_RCNN. A keras/tensorflow implementation of maskrcnn

Multimodal learning. Pytorch version of matterport.

Facebook detectron. Facebook super package that includes implementation of a wide range of image segmentation algorithms using Caffe2.

Pytorch detectron. Conversion of detectron to pytorch.