MaskAL - Active learning for Mask R-CNN in Detectron2

Summary

MaskAL is an active learning framework that automatically selects the most-informative images for training Mask R-CNN. By using MaskAL, it is possible to reduce the number of image annotations, without negatively affecting the performance of Mask R-CNN. Generally speaking, MaskAL involves the following steps:

1. Train Mask R-CNN on a small initial subset of a bigger dataset
2. Use the trained Mask R-CNN algorithm to make predictions on the unlabelled images of the remaining dataset
3. Select the most-informative images with a sampling algorithm
4. Annotate the most-informative images, and then retrain Mask R-CNN on the most informative-images
5. Repeat step 2-4 for a specified number of sampling iterations
   
   

The figure below shows the performance improvement of MaskAL on our dataset. By using MaskAL, the performance of Mask R-CNN improved more quickly and therefore 1400 annotations could be saved (see the black dashed line):

MaskAL instruction video

Installation

Linux/Ubuntu: INSTALL.md

Windows: INSTALL_Windows.md

Data preparation and training

Split the dataset in a training set, validation set and a test set. It is not required to annotate every image in the training set, because MaskAL will select the most-informative images automatically.

1. From the training set, a smaller initial dataset is randomly sampled (the dataset size can be specified in the maskAL.yaml file). The images that do not have an annotation are placed in the annotate subfolder inside the image folder. You first need to annotate these images with LabelMe (json), V7-Darwin (json), Supervisely (json) or CVAT (xml) (when using CVAT, export the annotations to LabelMe 3.0 format). Refer to our annotation procedure: ANNOTATION.md
2. Step 1 is repeated for the validation set and the test set (the file locations can be specified in the maskAL.yaml file).
3. After the first training iteration of Mask R-CNN, the sampling algorithm selects the most-informative images (its size can be specified in the maskAL.yaml file).
4. The most-informative images that don't have an annotation, are placed in the annotate subfolder. Annotate these images with LabelMe (json), V7-Darwin (json), Supervisely (json) or CVAT (xml) (when using CVAT, export the annotations to LabelMe 3.0 format).
5. OPTIONAL: it is possible to use the trained Mask R-CNN model to auto-annotate the unlabelled images to further reduce annotation time. Activate auto_annotate in the maskAL.yaml file, and specify the export_format (currently supported formats: 'labelme', 'cvat', 'darwin', 'supervisely').
6. Step 3-5 are repeated for several training iterations. The number of iterations (loops) can be specified in the maskAL.yaml file.

Please note that MaskAL does not work with the default COCO json-files of detectron2. These json-files contain all annotations that are completed before the training starts. Because MaskAL involves an iterative train and annotation procedure, the default COCO json-files lack the desired format.

How to use MaskAL

Open a terminal (Ctrl+Alt+T):

(base) user@computer:~$ cd maskal
(base) user@computer:~/maskal$ conda activate maskAL
(maskAL) user@computer:~/maskal$ python maskAL.py --config maskAL.yaml

Change the following settings in the maskAL.yaml file:

Description of the other settings in the maskAL.yaml file: MISC_SETTINGS.md

Please refer to the folder active_learning/config for more setting-files.

Other software scripts

Use a trained Mask R-CNN algorithm to auto-annotate unlabelled images: auto_annotate.py

Example syntax (auto_annotate.py):

python auto_annotate.py --img_dir datasets/train --network_config COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x.yaml --classes healthy damaged matured cateye headrot --conf_thres 0.5 --nms_thres 0.2 --weights_file weights/broccoli/model_final.pth --export_format supervisely --supervisely_meta_json datasets/meta.json

Troubleshooting

See TROUBLESHOOTING.md

Citation

See our research article on arXiv and Computers and Electronics in Agriculture for more information:

@misc{blok2021active,
  title = {Active learning with MaskAL reduces annotation effort for training Mask R-CNN},
  author = {Blok, Pieter M. and Kootstra, Gert and Elghor, Hakim Elchaoui and Diallo, Boubacar and van Evert, Frits K. and van Henten, Eldert J.},
  year = {2021},
  doi = {10.48550/ARXIV.2112.06586},
  url = {https://arxiv.org/abs/2112.06586},
  publisher = {arXiv},
  copyright = {Creative Commons Attribution Share Alike 4.0 International},
  keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences}
}

@article{blok2022106917,
  title = {Active learning with MaskAL reduces annotation effort for training Mask R-CNN on a broccoli dataset with visually similar classes},
  author = {Pieter M. Blok and Gert Kootstra and Hakim Elchaoui Elghor and Boubacar Diallo and Frits K. {van Evert} and Eldert J. {van Henten}},
  journal = {Computers and Electronics in Agriculture},
  volume = {197},
  pages = {106917},
  year = {2022},
  issn = {0168-1699},
  doi = {10.1016/j.compag.2022.106917},
  url = {https://www.sciencedirect.com/science/article/pii/S0168169922002344}
}

License

Our software was forked from Detectron2 (https://github.com/facebookresearch/detectron2). As such, the software will be released under the Apache 2.0 license.

Acknowledgements

The uncertainty calculation methods were inspired by the research of Doug Morrison:
https://nikosuenderhauf.github.io/roboticvisionchallenges/assets/papers/CVPR19/rvc_4.pdf

Two software methods were inspired by the work of RovelMan:
https://github.com/RovelMan/active-learning-framework

MaskAL uses the Bayesian Active Learning (BaaL) software:
https://github.com/ElementAI/baal

Contact

MaskAL is developed and maintained by Pieter Blok.