[Feature Request]: CLIP Driven Universal Model

[tangy5]

Contrastive Language-Image Pre-training (CLIP) Driven Models and Partially Supervised Learning for Medical Image Segmentation

This issue is to discuss adding the CLIP-Driven Universal Model Features to MONAI.

Potential assignee: @tangy5

CLIP-Driven Universal Model

Key features

The implementation will bring several new feature as follows:

1. Universal Model: one model to detect and segment all abdominal organs and all types of tumors (Liver tumor, kidney tumor, Lung nodule, Pancreas tumor, hepatic vessel tumor, colon tumor).
2. Language model (CLIP) and text-driven embeddings boost medical image analysis.
3. Training Partial labelled datasets.
4. Incremental learning: Users can continue to train new segmentation classes using the current trained model without catastrophic forgetting.

⏳ Dataset: The Universal Model is trained with following datasets

• 01 Multi-Atlas Labeling Beyond the Cranial Vault - Workshop and Challenge (BTCV)
• 02 Pancreas-CT TCIA
• 03 Combined Healthy Abdominal Organ Segmentation (CHAOS)
• 04 Liver Tumor Segmentation Challenge (LiTS)
• 05 Kidney and Kidney Tumor Segmentation (KiTS)
• 06 Liver segmentation (3D-IRCADb)
• 07 WORD: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from CT image
• 08 AbdomenCT-1K
• 09 Multi-Modality Abdominal Multi-Organ Segmentation Challenge (AMOS)
• 10-15 Decathlon (Liver, Lung, Pancreas, HepaticVessel, Spleen, Colon)
• 16 CT volumes with multiple organ segmentations (CT-ORG)
• 17 13 AbdomenCT 12organ

Implementation plans

• [ ] Transformations (pre-processing) for partial labelled datasets: “PartialLabelTransfer”, etc
• [ ] Segmentation backbone with CLIP embedding, text-driven segmentor: plug-and-play CLIP embedding and text encoder.
• [ ] Tutorial for training and inference of Universal Model.
• [ ] Tutorial for demonstrating partial supervised learning and incremental learning.
• [ ] Model release: Bundle for Model Zoo for publishing the trained universal model to segment all types of tumours and abdominal organs.

More Details of the Feature Methodology:

1. Universal Model:

2. CLIP Driven and text-driven segmentor:

3. Partial Supervised Learning:

4. Incremental Leraning:

Detailed steps of implantation will provide after open discussion.

Welcome all suggestions and comments!

@ljwztc @MrGiovanni

[Da1daidaidai]

looking forward for updating

[tangy5]

looking forward for updating

Thanks, very appreciative, will continue to working on this, and provide useful API for CLIP related integration. Welcome comments and suggestions.