Harnessing Diffusion Models for Visual Perception with Meta Prompts

Paper

Harnessing Diffusion Models for Visual Perception with Meta Prompts,
Qiang Wan, Zilong Huang, Bingyi Kang, Jiashi Feng, Li Zhang

📸 Release

• ⏳ Pose estimation training code and model.
• Jan. 31th, 2024: Release semantic segmentation training code and model.
• Jan. 6th, 2024: Release depth estimation training code and model.

Installation

Clone this repo, and run

sh install.sh

Download the checkpoint of stable-diffusion (we use v1-5 by default) and put it in the checkpoints folder.

Depth Estimation with meta prompts

MetaPrompts obtains 0.223 RMSE on NYUv2 depth estimation benchmark and 1.929 RMSE on KITTI Eigen split, establishing the new state-of-the-art.

Please check depth.md for detailed instructions on training and inference.

Semantic segmentation with meta prompts

MetaPrompts obtains 56.8 mIoU on ADE20K semantic segmentation benchmark and 87.3 mIoU on CityScapes, establishing the new state-of-the-art.

Please check segmentation.md for detailed instructions on training and inference.

License

MIT License

Acknowledgements

This code is based on stable-diffusion, mmsegmentation, LAVT, VPD, ViTPose, mmpose, and MIM-Depth-Estimation.

BibTeX

If you find our work useful in your research, please consider citing:

@article{wan2023harnessing,
  title={Harnessing Diffusion Models for Visual Perception with Meta Prompts},
  author={Wan, Qiang and Huang, Zilong and Kang, Bingyi and Feng, Jiashi and Zhang, Li},
  journal={arXiv preprint arXiv:2312.14733},
  year={2023}
}