Prompting Vision Foundation Models for Pathology Image Analysis
Updates
- TO DO: Add instruction
- [06/2024] Code Released.
Introduction
This is the github repository of Prompting Vision Foundation Models for Pathology Image Analysis. In this paper, we propose Quantitative Attribute-based Prompting (QAP), a novel prompting method specifically for liver pathology image analysis. QAP is based on two quantitative attributes, namely K-function-based spatial attributes and histogram-based morphological attributes, which are aimed for quantitative assessment of tissue states. Moreover, a conditional prompt generator is designed to turn these instance-specific attributes into visual prompts.
The framework of the proposed QAP model:
Enhanced Interpretability
Image samples with its attention map and attribute significance histogram when identifying specific histological findings. Our method enhances interpretability by visually representing the decision-making process through attention maps and attribute significance histograms. a. The structures the model focuses on; b. The attributes of structures the model focus on.
Boost Diagnosis Performance
We further explore learning various prompts. The tissue structure segments provide more informative cues compared to task-agnostic visual prompts learned from randomly initialized vectors. Additionally, the quantitative attributes obtained from summarizing the statistical information about tissue structures are more explicit. Using prompts learned conditioned on explicit cues can enhance the learning process and improve performance.
Citation
If you find this work useful for your research, please kindly cite our paper:
@inproceedings{yin2024prompting,
title={Prompting Vision Foundation Models for Pathology Image Analysis},
author={Yin, Chong and Liu, Siqi and Zhou, Kaiyang and Wong, Vincent Wai-Sun and Yuen, Pong C},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={11292--11301},
year={2024}
}