EKAGen

Code for CVPR2024 paper: "Instance-level Expert Knowledge and Aggregate Discriminative Attention for Radiology Report Generation". Shenshen Bu, Taiji Li, Yuedong Yang, Zhiming Dai. [Video]

Abstract: Automatic radiology report generation can provide substantial advantages to clinical physicians by effectively reducing their workload and improving efficiency. Despite the promising potential of current methods, challenges persist in effectively extracting and preventing degradation of prominent features, as well as enhancing attention on pivotal regions. In this paper, we propose an Instance-level Expert Knowledge and Aggregate Discriminative Attention framework for radiology report generation. We convert expert reports into an embedding space and generate comprehensive representations for each disease, which serve as Preliminary Knowledge Support (PKS). To prevent feature disruption, we select the representations in the embedding space with the smallest distances to PKS as Rectified Knowledge Support (RKS). Then, EKAGen diagnoses the diseases and retrieves knowledge from RKS, creating Instance-level Expert Knowledge (IEK) for each query image, boosting generation. Additionally, we introduce Aggregate Discriminative Attention Map (ADM), which uses weak supervision to create maps of discriminative regions that highlight pivotal regions. For training, we propose a Global Information Self-Distillation (GID) strategy, using an iteratively optimized model to distill global knowledge into EKAGen. Extensive experiments and analyses on IU X-Ray and MIMIC-CXR datasets demonstrate that EKAGen outperforms previous state-of-the-art methods.

Get Started

1) Requirement

• Python 3.8.13
• Pytorch 1.9.0
• Torchvision 0.10.0
• CUDA 11.8
• NVIDIA RTX 4090

2) Data Preparation

MIMIC-CXR

• You must be a credential user defined in PhysioNet to access the data.
• Download chest X-rays from MIMIC-CXR-JPG and reports from MIMIC-CXR Database.

IU X-Ray

• You can download the processed reports and images for IU X-Ray by Chen et al. from R2GenCMN.

3) Download Model Weights and Knowledge Base

• Download the following model weights:	Model	Publicly Available
  DiagnosisBot	diagnosisbot.pth
  Generate ADM Model Weight	MIMIC_best_weight.pth
  IU X-Ray Teacher Model	iu_t_model.pth
  MIMIC-CXR Teacher Model	mimic_t_model.pth

• Download the following knowledge base and attention maps:	Item	Publicly Available
  IU X-Ray Knowledge Base	knowledge_prompt_iu.pkl
  MIMIC-CXR Knowledge Base	knowledge_prompt_mimic.pkl
  IU X-Ray ADM	iu_mask.tar.gz
  MIMIC-CXR ADM	mimic_mask.tar.gz

4) Training

IU X-Ray

bash train_iu.sh

MIMIC-CXR

bash train_mimic.sh

5) Inference

You can download our trained models for inference from IU X-Ray and MIMIC-CXR.

IU X-Ray

bash test_iu.sh

MIMIC-CXR

bash test_mimic.sh

Citation

If you find this work useful in your research, please cite:

@InProceedings{Bu_2024_CVPR,
    author    = {Bu, Shenshen and Li, Taiji and Yang, Yuedong and Dai, Zhiming},
    title     = {Instance-level Expert Knowledge and Aggregate Discriminative Attention for Radiology Report Generation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {14194-14204}
}

Contact Information

If you have any suggestions or questions, you can contact us by: bushsh@mail2.sysu.edu.cn. Thank you for your attention!