Finetune ClipSeg model

[sleeping4cat]

Feature request

Quite recently, I was exploring zero-shot classification to segment medical images. And it looks quite promising. I stumbled upon ClipSeg a few days ago and it looked wonderful and just well-suited for my work. Unfortunately, I couldn't find any tutorials or notebooks that showed how to perform fine-tuning on ClipSeg model.

I am assuming, we have to train the decoder with a dataset containing binary classification images of cells and their corresponding masks and a text description. Unfortunately, a bit confused. is there any tutorials/resources anyone could suggest on this topic? Cuz I couldn't none.

Motivation

ClipSeg shows a lot of potential than SAM (Segment Anything Model). Unfortunately, there's no fine-tuning script neither instructions on How to prepare the dataset? which is very frustrating. Will love some help from the community.

And another point, Zero shot classification looks a way lot better option with fine-tuning than training a model like U-Net, R-CNN and others from scratch while you have very few images and don't have much room to play around with.

Your contribution

I could provide a PR on my LinkedIn, where I have a lot of AI experts as my connections and then I contribute in the programming as well.

[sgugger]

cc @alaradirik for information.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sleeping4cat]

@sgugger Can you provide some insights/help/update to my requested feature?

[sgugger]

cc @amyeroberts and @rafaelpadilla