DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation (ICCV 2019)

paper.pdf

Overview

Requirements:

• PyTorch 1.1.0
• matplotlib 3.0.2
• not in requirements.txt:
  • cython
  • torchvision 0.2.2 or 0.3.0 if cuda10.0
  • pycocotools # (2.0-py3.7-linux-x86_64)
  • maskrnn-benchmark
  • pyyaml yacs
  • opencv-python scikit-image
  • easydict prettytable lmdb tabulate

Installation

• Follow INSTALL.md

Data

YouTube-VOS

• Download the YouTube-VOS dataset from their website. Please note that our code is trained and tested only on YouTube-VOS dataset for 2018 version. There is a newer version released 2019 but it is not tested.

• We recommend to symlink the path to the youtube dataset to datasets/ as follows

  cd datasets 
  ln -s path/to/youtubeVOS youtubeVOS 

• The files structure should look like:

  DMM/datasets 
     ├── youtubeVOS
     │       ├── train 
     │       │      ├── JPEGImages
     │       │      │        ├── ... 
     │       │      ├── Annotations 
     │       │      │        ├── ... 
     │       ├── valid 
     │       │      ├── JPEGImages
     │       │      │        ├── ... 
     │       │      ├── Annotations 
     │       │      │        ├── ... 
     │       ├── train_testdev_ot (optional)
     │       │      ├── JPEGImages
     │       │      │        ├── ... 
     │       │      ├── Annotations 
     │       │      │        ├── ... 

• the train_testdev_ot data can be downloaded from link

Prepare proposals

Option1: Download the extracted file

for evaluation

• To eval DMMnet on youtubeVOS with the fine-tuned proposal net, use the propsoals generated by our fine-tuned Mask R-CNN model:
  • proposals-train-val

    mkdir -p experiments/proposals/ 
    cd experiments/proposals/  
    wget https://www.cs.toronto.edu/~xiaohui/dmm/proposals/proposals_ytb_train.tar.gz
    tar xzf proposals_ytb_train.tar.gz 

for training

• To train the DMMnet on youtubeVOS train-train split, need to prepare 1. proposals for both train-train and train-val split extracted by coco pretrained X101 Mask R-CNN model

• proposals can be downloaded:

  • proposals_coco81 (train-train, train-val and testdev-online-training)

    mkdir -p experiments/proposals/  
    cd experiments/proposals/ 
    wget http://www.cs.toronto.edu/~xiaohui/dmm/proposals/feature_coco81.tar.gz
    tar xzf feature_coco81.tar.gz 

• preprocess the proposals for training DMM:

  python src/tools/reduce_pth_size_by_videos.py  experiments/proposals/coco81/inference/youtubevos_train3k_meta/predictions.pth  train 50
  python src/tools/reduce_pth_size_by_videos.py  experiments/proposals/coco81/inference/youtubevos_val200_meta/predictions.pth  trainval 50
  python src/tools/reduce_pth_size_by_videos.py  experiments/proposals/coco81/inference/youtubevos_testdev_online_meta/predictions.pth  train_testdev_ot 90 

• The files structure should look like:

  DMM/experiments
     ├── propnet
     │     ├── join_ytb_bin
     │     │       ├── model_0172500.pth 
     │     ├── online_ytb
     │     │       ├── model_0225000.pth 
     ├── dmmnet 
     │     ├── ytb_255_50_matchloss_epo13
     │     │       ├── epo13_iter01640
     │     ├── ytb_255_50
     │     │       ├── epo08_iter01640
     │     ├── online_ytb
     │     │       ├── epo101 
     ├── proposals 
     │     ├── coco81 
     │     │     ├── inference
     │     │     │       ├── youtubevos_train3k_meta (optional)
     │     │     │       ├── youtubevos_val200_meta
     │     │     │       ├── youtubevos_testdev_online_meta (optional)
     │     ├── ytb_train 
     │     │     ├── inference
     │     │     │       ├── youtubevos_val200_meta
     │     ├── ytb_ot 
     │     │     ├── inference
     │     │     │       ├── youtubevos_testdev_meta

Option2: extract the proposals

• The model trained on youtubeVOS dataset can be found in MODEL_ZOO.md
• The scripts used to extract proposals from the trained model can be found in scripts/extract/

Training

• Train DMMnet on youtubeVOS:

  sh scripts/train/train_101.sh 
  # or scripts/train/train_50.sh # for resnet 50 mode  

Online training

Train DMMnet on the first frame of validation set,

• first download the preprocessed data used for online training from here, extract the data and put/link the extracted folder as /PATH/TO/datasets/youtubeVOS/train_testdev_ot

• prepare proposal, check the Section: Prepare proposals - for training

• get the DMMnet trained on train-train set for 1 epoch from here and put it under experiments/dmmnet/

• start online training

  sh scripts/train/train_online.sh # it takes ~0.17h for one epoch

Evaluation

• Evaluate DMMnet on trainval split:

  • will need the trained model and the extracted train-val proposal:

    
    cd ./experiments/dmmnet/
    wget http://www.cs.toronto.edu/~xiaohui/dmm/models/dmmnet_ytb_255_50_matchloss_epo13.tar.gz 
    tar xzf dmmnet_ytb_255_50_matchloss_epo13.tar.gz 
    wget http://www.cs.toronto.edu/~xiaohui/dmm/models/dmmnet_ytb_255_50.tar.gz 
    tar xzf dmmnet_ytb_255_50.tar.gz
    cd ../../ 

  cd ./experiments/proposals/ wget http://www.cs.toronto.edu/~xiaohui/dmm/proposals/proposals_ytb_train.tar.gz tar xzf proposals_ytb_train.tar.gz cd ../../

  
  - run `sh scripts/eval/eval_r50.sh` 
  - compute the J and F score by `sh scripts/metric/full_eval.sh /PATH/TO/OUTPUT/merged/`
  expected results:

• Evaluate online-trained DMMnet:
  • Download proposals extracted by online-trained proposal net:

    cd ./experiments/proposals/
    wget http://www.cs.toronto.edu/~xiaohui/dmm/proposals/proposals_ytb_ot.tar.gz 
    tar xzf proposals_ytb_ot.tar.gz 
    cd ../../

  • Download model:

    cd experiments/dmmnet/
    wget http://www.cs.toronto.edu/~xiaohui/dmm/models/dmmnet_online_ytb.tar.gz 
    tar xzf dmmnet_online_ytb.tar.gz 
    cd ../../

  • run scripts/eval/eval_testdev.sh
  • prepare the submission data with scripts/submit.sh and submit to the server, expected resules: G mean = 0.579

part of the code is from https://github.com/imatge-upc/rvos