Unpair-Seg: Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision

This repo contains the code for our paper UnpairSeg. It is a weakly supervised open-vocabulary segmentation framework that leverages unpaired mask-text pairs.

Now, we release the inference code and checkpoints for stage one training.

Installation

• Linux with Python ≥ 3.10
• PyTorch ≥ 2.1 and torchvision that matches the PyTorch installation. Install them together at pytorch.org to make sure of this. Note, please check PyTorch version matches that is required by Detectron2.
• Detectron2: follow Detectron2 installation instructions.
• OpenCV is optional but needed by demo and visualization
• please check install.sh for other dependencies

Inference

The part provides a brief introduction of the usage of Unpair-Seg. Please download the checkpoint of stage one training. We provide ./demo/inference.py for point-promptable segmentation. Run it with:

cd demo/
python inference.py \
    -c ../configs/S1_point_seg.yaml \
    -i ../images/*.jpg \
    --opt MODEL.WEIGHTS stage1.pth

We also provide some test images under ./images/.

If you use this codebase, or otherwise found our work valuable, please cite:

@article{wang2024open,
  title={Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision},
  author={Wang, Zhaoqing and Xia, Xiaobo and Chen, Ziye and He, Xiao and Guo, Yandong and Gong, Mingming and Liu, Tongliang},
  journal={arXiv preprint arXiv:2402.08960},
  year={2024}
}