Shatter and Gather: Learning Referring Image Segmentation with Text Supervision

This repository contains the official source code for our paper:

Shatter and Gather: Learning Referring Image Segmentation with Text Supervision Dongwon Kim¹, Namyup Kim¹, Cuiling Lan², and Suha Kwak^1
1POSTECH CSE, ²Microsoft Research Asia
ICCV, Paris, 2023.

Dataset setup

Setting

• Download the MS COCO images are under data/coco/images/train2014/
• Download refer under external. Then strictly follow the Setup and Download section. Also put the refer folder in PYTHONPATH
• Download the MS COCO API also under external (i.e. external/coco/PythonAPI/pycocotools)

Data preparation

python build_batches.py -d Gref -t train 
python build_batches.py -d Gref -t val 
python build_batches.py -d unc -t train 
python build_batches.py -d unc -t val 
python build_batches.py -d unc -t testA 
python build_batches.py -d unc -t testB 
python build_batches.py -d unc+ -t train 
python build_batches.py -d unc+ -t val 
python build_batches.py -d unc+ -t testA 
python build_batches.py -d unc+ -t testB 

Final ./data directory structure

./data
├─ refcoco
│   ├─ Gref
│   │   ├─ train_batch
│   │   │   ├─ Gref_train_0.npz
│   │   │   ├─ Gref_train_1.npz
│   │   │   └─ ...
│   │   ├─ train_image
│   │   ├─ train_label
│   │   ├─ val_batch
│   │   ├─ val_image
│   │   └─ val_label
│   ├─ unc
│   │   └─ ...
│   └─ unc+
│       └─ ...
├─ phrasecut
│   └─ images
│      ├─ refer_train_ris.json
│      ├─ refer_val_ris.json
│      └─  refer_test_ris.json
├─ Gref_emb.npy
├─ referit_emb.npy
├─ vocabulary_Gref.txt
└─ vocabulary_referit.txt

Environment setup

• Python 3.10.9
• PyTorch 1.13.1+cu117

Instructions:

conda create -n sag python=3.10 -y
conda activate sag
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu11
pip install einops tqdm wandb transformers
pip install matplotlib timm opencv-python

Train & eval

sh ./train_eval_gref.sh # Gref
sh ./train_eval_unc.sh # UNC
sh ./train_eval_unc+.sh # UNC+

Acknowledgement

Parts of our codes are adopted from the following repositories.

• https://github.com/lucidrains/perceiver-pytorch
• https://github.com/kdwonn/DivE

Dataset Setup instruction is from TF-phrasecut-public repository.