MVViT

Multi-View Vision Transformers for ICPR 2022

Overview

This is the code for Multi-view Vision Transformers (MVViT), a Transformer-based network that combines features from multiple views to improve object detection. It is based on MMDetection 2.15.1, so in order to use it simply clone this repository in the root folder where MMDetection was installed. All example commands in this repository are considered to run from the root directory of MMDetection.

Dependencies

• Python 3.8
• Pytorch 1.7.1
• torchvision 0.8.2
• MMDetection 2.15.1
• pycocotools 2.0.2
• numpy 1.20
• matplotlib 3.4.2
• opencv-python 4.5
• tqdm 4.62
• pillow 8.3.1

Data preparation

Datasets must be in COCO format. A different annotations file must be provided for each view, where the image id is the same for images from the same scene. These files are used in the config files.

A script to convert the Wildtrack dataset annotations into multi-view coco annotations is provided in tools/wildtrac2mvcoco.py:

python MVViT/tools/wildtrack2mvcoco.py --anns-dir /path/to/wildtrack/dataset/annotation/annotations_positions --output-dir /path/to/output/dir

Usage

Training and testing is done as usual in the MMDetection framework. New fields are added in the config files for multi-view YOLOv3 and Deformable DETR detectors, which are described next.

To integrate MVViT to YOLOv3, the following classes were added:

• MVViTDarknet
• YOLOV3MVHead

For Deformable DETR, the new classes are:

• MVViTResnet
• MVChannelMapper
• YOLOV3MVHead

Additionally, new datasets and pipelines focused in handling multi-view datasets are provided.

Config files

For multi-view data loading, the following custom imports must be added:

'MVViT.datasets.pipelines.formatting',
'MVViT.datasets.pipelines.loading',
'MVViT.datasets.pipelines.test_time_aug',
'MVViT.datasets.pipelines.transforms',
'MVViT.datasets.coco_mv',
'MVViT.datasets.custom_mv'

The following changes must be done to the train and test pipelines (currently, RandomFlip is not supported):

• LoadImageFromFile -> LoadMVImagesFromFile
• LoadAnnotations -> LoadMVAnnotations
• Pad -> MVPad
• Resize -> MVResize
• Normalize -> MVNormalize
• DefaultFormatBundle -> MVFormatBundle
• MultiScaleFlipAug -> MVMultiScaleFlipAug
• ImageToTensor -> MVImageToTensor

Additionally, dataset_type must be set to MVCocoDataset and the ann_file must be changed to ann_files and provide an ordered list of annotation files, each file corresponding to a view.

YOLOv3 Detection

YOLOv3 uses the following custom imports:

'MVViT.models.backbones.mvvit_darknet',
'MVViT.models.dense_heads.mv_yolo_head'

The backbone must be changed from Darknet to MVViTDarknet with the following new fields:

• combination_block: where in the backbone the feature maps are combined via MVViT. If the passed value is less than -1, MVViT is not used. A value of -1 indicates a combination in the image space. For values greater than -1, the combination block is the Darknet block before using the MVViT module. Default: -2
• nhead: number of heads used in MVViT. Default: 8
• num_decoder_layers: the number of decoders used in MVViT. Default: 1.
• positional_encoding: boolean indicating if positional encoding is used in MVViT. Default: True
• views: number of views.

Also, YOLOV3Head must be changed to YOLOV3MVHead.

Deformable DETR Detection

Similar to YOLOv3, the following custom imports must be added:

'MVViT.models.backbones.mvvit_resnet',
'MVViT.models.necks.mv_channel_mapper',
'MVViT.models.dense_heads.mv_deformable_detr_head',

The backbone must be changed from ResNet to MVViTResNet with the following new fields:

• combination_block: where in the backbone the feature maps are combined via MVViT. If the passed value is less than 0, MVViT is not used. A value of 0 indicates a combination in the image space. For values greater than 0, the combination block is the ResNet block before using the MVViT module. Default: -1
• nhead: number of heads used in MVViT. Default: 8
• num_decoder_layers: the number of decoders used in MVViT. Default: 1.
• positional_encoding: boolean indicating if positional encoding is used in MVViT. Default: True
• views: number of views.

Also, ChannelMapper must be changed to MVChannelMapper and DeformableDETRHead to MVDeformableDETRHead.

Training

Training is carried out with the standard MMDetection script

python tools/train.py /path/to/config/file

Testing

We provide a multi-view testing script, similar to MMDetection file.

python MVViT/tools/test.py /path/to/config/file /path/to/checkpoint [same options for MMDetection test...]

Results

Citation

@InProceedings{isaac-medina2022mvvit,
  author={Isaac-Medina, Brian K. S. and Willcocks, Chris G. and Breckon, Toby P.},
  booktitle={2022 26th International Conference on Pattern Recognition (ICPR)}, 
  title={Multi-view Vision Transformers for Object Detection}, 
  year={2022},
  pages={4678-4684},
  doi={10.1109/ICPR56361.2022.9956443}
  }