Contrastive Language-Image Pre-training (CLIP) Driven Models and Partially Supervised Learning for Medical Image Segmentation
This issue is to discuss adding the CLIP-Driven Universal Model Features to MONAI.
Potential assignee: @tangy5
CLIP-Driven Universal Model
Key features
The implementation will bring several new feature as follows:
Universal Model: one model to detect and segment all abdominal organs and all types of tumors (Liver tumor, kidney tumor, Lung nodule, Pancreas tumor, hepatic vessel tumor, colon tumor).
Language model (CLIP) and text-driven embeddings boost medical image analysis.
Training Partial labelled datasets.
Incremental learning: Users can continue to train new segmentation classes using the current trained model without catastrophic forgetting.
⏳ Dataset: The Universal Model is trained with following datasets
Contrastive Language-Image Pre-training (CLIP) Driven Models and Partially Supervised Learning for Medical Image Segmentation
This issue is to discuss adding the CLIP-Driven Universal Model Features to MONAI.
Potential assignee: @tangy5
CLIP-Driven Universal Model
Key features
The implementation will bring several new feature as follows:
⏳ Dataset: The Universal Model is trained with following datasets
Implementation plans
More Details of the Feature Methodology:
Universal Model:![Screenshot from 2023-01-03 12-09-23](https://user-images.githubusercontent.com/58751975/210440352-50b68a10-69f5-48d5-bbca-c100ccd680a0.png)
CLIP Driven and text-driven segmentor:![Screenshot from 2023-01-03 12-10-09](https://user-images.githubusercontent.com/58751975/210440462-4607db99-72a3-4908-ad8e-68e7186282d2.png)
Partial Supervised Learning:![Screenshot from 2023-01-03 12-04-46](https://user-images.githubusercontent.com/58751975/210440515-2d58f5cf-134f-4c53-ba77-2fefde51644e.png)
Incremental Leraning:
Detailed steps of implantation will provide after open discussion.
Welcome all suggestions and comments!
@ljwztc @MrGiovanni