MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine

MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine
Yunfei Xie*, Ce Zhou*, Lang Gao*, Juncheng Wu*, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, Yuyin Zhou

📢 Breaking News

[🆕💥 August 31, 2024] Detailed tutorial for deploying MedTrinity now available at HuggingFace. We apologize for any previous inconvenience.
[📄💥 August 7, 2024] Our arXiv paper is released.
[💾 July 21, 2024] Full dataset released.
[💾 June 16, 2024] Demo dataset released.

Star 🌟 us if you think it is helpful!!

🚀 Dataset

Dataset construction pipeline

1) Data processing: extracting essential information from collected data, including metadata integration to generate coarse captions, ROI locating, and medical knowledge collection. 2) Multigranular textual description generation: using this information to prompt MLLMs to generate fine-grained captions.

Statistical overview of MedTrinity-25M

Statistics of MedTrinity-25M

You can view detailed statistics of MedTrinity-25M from this link.

Note: sometimes a single image contains multiple biological structures. The data only reflect the number of samples in which a specific biological structure is present.

Dataset Download

Dataset	🤗 Huggingface Hub
MedTrinity-25M	UCSC-VLAA/MedTrinity-25M

🏆 Results

💬 Quick Start

Install

Using Linux system,

Clone this repository and navigate to the folder

git clone https://github.com/UCSC-VLAA/MedTrinity-25M.git

Install Package

conda create -n llava-med++ python=3.10 -y
conda activate llava-med++
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

Install additional packages for training cases

pip install -e ".[train]"
pip install flash-attn --no-build-isolation
pip install git+https://github.com/bfshi/scaling_on_scales.git
pip install multimedeval

Upgrade to latest code base

git pull
pip install -e .

# if you see some import errors when you upgrade,
# please try running the command below (without #)
# pip install flash-attn --no-build-isolation --no-cache-dir

🤖 Model-Zoo

The following table provides an overview of the available models in our zoo. For each model, you can find links to its Hugging Face page or Google drive folder.

Model Name	Link	Summary
LLaVA-Med++ (VQA-RAD)	Google Drive	Pretrained on LLaVA-Med Data and MedTrinity-25M (specifically the VQA-RAD training set subset), finetuning on VQA-RAD training set.
LLaVA-Med++ (SLAKE)	Google Drive	Pretrained on LLaVA-Med Data and MedTrinity-25M (specifically the SLAKE training set subset), finetuning on SLAKE training set.
LLaVA-Med++ (PathVQA)	Google Drive	Pretrained on LLaVA-Med Data and MedTrinity-25M (specifically the PathVQA training set subset), finetuning on PathVQA training set.
LLaVA-Med-Captioner	Hugging Face	Captioner for generating multigranular annotations fine-tuned on MedTrinity-Instruct-200K (Coming soon).

Train and Eval LLaMA-Med++

First, you need to download the base model LLaVA-Meta-Llama-3-8B-Instruct-FT-S2 and download the stage1 and stage2 datasets in the LLaVA-Med.

Pre-train


# stage1 training
cd MedTrinity-25M
bash ./scripts/med/llava3_med_stage1.sh

stage2 training

bash ./scripts/med/llava3_med_stage2.sh

2. Finetune
```bash
cd MedTrinity-25M
bash ./scripts/med/llava3_med_finetune.sh

Eval
First, you need to download corresponding weight from Model-Zoo and change the path in evaluation script. Then run:
```
cd MedTrinity-25M
bash ./scripts/med/llava3_med_eval_batch_vqa_rad.sh
```

📜 Citation

If you find MedTrinity-25M useful for your research and applications, please cite using this BibTeX:

@misc{xie2024medtrinity25mlargescalemultimodaldataset,
      title={MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine}, 
      author={Yunfei Xie and Ce Zhou and Lang Gao and Juncheng Wu and Xianhang Li and Hong-Yu Zhou and Sheng Liu and Lei Xing and James Zou and Cihang Xie and Yuyin Zhou},
      year={2024},
      eprint={2408.02900},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2408.02900}, 
}

🙏 Acknowledgement

We thank the Microsoft Accelerate Foundation Models Research Program, the OpenAI Researcher Access Program, TPU Research Cloud (TRC) program, Google Cloud Research Credits program, AWS Cloud Credit for Research program, and Lambda Cloud for supporting our computing needs.
Thanks for the codebase of LLaVA-pp, LLaVA-Med and LLaVA we built upon, and our base model LLaVA-Meta-Llama-3-8B-Instruct-FT-S2 that has the amazing language capabilities!

UCSC-VLAA / MedTrinity-25M

readme