This is the official Pytorch implementation of our CVPR 2024 paper (Highlight) "Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning".
CUDA 11.5
Python 3.8
Pytorch 1.11.0
CuDNN 8.3.2.44
Fine-tuning data
RICORD dataset: Download the MIDRC-RICORD-1A dataset and MIDRC-RICORD-1B dataset. The folder structure of the dataset should be like
dataset/RICORD_nii/ ├── MIDRC-RICORD-1A ├── MIDRC-RICORD-1B
Preprocess/MIMIC_CXR_JPG_Preprocess.py
to pre-process the MIMC-CXR dataset.Preprocess/DL_save_nifti.py
(from downloaded files) to transfer the PNG image to the nii.gz form.Preprocess/re_spacing_ITK.py
to resample CT volumes.Preprocess/splitting_to_patches.py
to extract about 125k sub-volumes, and the pre-processed dataset will be saved in DL_patches_v2/
.Preprocess/DeepLesion_Resize.py
to resize images.Preprocess/ADNI_Resize.py
to resize images.Preprocess/ADNI_split_slice.py
to extract about 59k sub-volumes.Preprocess/TCGA_Preprocess.py
to pre-process seven TCGA datasets.Preprocess/QaTav2.py
to pre-process.Preprocess/RICORD.py
to pre-process. Data Splits can be obtained from /Downstream/Dim_3/RICORD/data_split
.Preprocess/VSeg.py
to pre-process.Preprocess/GlaS.py
to pre-process.sh run_ssl.sh
for pre-training (4 GPUs with 24G. Before running it, you need to modify some addresses.)sh run_ds.sh
for fine-tuning. (one GPU with 11G. Before running it, you need to modify some addresses.)If this code is helpful for your study, please cite:
@article{ye2024medcoss,
title={Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning},
author={Ye, Yiwen and Xie, Yutong and Zhang, Jianpeng and Chen, Ziyang and Wu, Qi and Xia, Yong},
booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
pages={11114-11124},
year={2024},
}
The whole framework is based on MAE, Uni-Perceiver, and MGCA.
Yiwen Ye (ywye@mail.nwpu.edu.cn)