google / diffseg

DiffSeg is an unsupervised zero-shot segmentation method using attention information from a stable-diffusion model. This repo implements the main DiffSeg algorithm and additionally includes an experimental feature to add semantic labels to the masks based on a generated caption.
https://sites.google.com/corp/view/diffseg
MIT License
267 stars 21 forks source link
computer-vision segmentation stable-diffusion

Diffuse, Attend, and Segment

This repo implements the DiffSeg segmentation method in the paper Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion.

 @article{tian2023diffuse,
  title={Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion},
  author={Tian, Junjiao and Aggarwal, Lavisha and Colaco, Andrea and Kira, Zsolt and Gonzalez-Franco, Mar},
  journal={arXiv preprint arXiv:2308.12469},
  year={2023}
}

Overview

DiffSeg is an unsupervised zero-shot segmentation method using attention information from a stable-diffusion model. This repo implements the main DiffSeg algorithm and additionally include an experimental feature to add semantic labels to the masks based on a generated caption.

More details can be found on the project page: https://sites.google.com/corp/view/diffseg/home

Create conda environment

Computation Requirement

DiffSeg Notebook

Please see the instructions in the diffseg.ipynb for running instructions.

Benchmarks

We benchmark the performance of DiffSeg on CoCo-Stuff-27 and Cityscapes. Please see instructions in benchmarks.ipynb.

Contributors