search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
132.05k stars 26.3k forks source link

Why there is no LlavaForSequenceClassification ? #31814

Open Zuhashaik opened 2 months ago

Zuhashaik commented 2 months ago

Feature request

Need of LlavaForSequenceClassification class in 'src/transformers/models/llava/modeling_llava.py'..

Motivation

If there is a LlavaForSequenceClassification class we can use decoder only models for classification tasks.

Your contribution

Allow me to contribute to this as I am already working on this model for mental health meme classification for the upcoming AAAI conference.

amyeroberts commented 2 months ago

Hi @Zuhashaik, thanks for opening this issue!

We only add specific task heads for models if there are official checkpoints available; the task is described in the original paper or there's a large community demand.

NielsRogge commented 2 months ago

Note that you could technically also use LlavaForConditionalGeneration by training the model to only spit out one classification token per image.

Zuhashaik commented 2 months ago

Note that you could technically also use LlavaForConditionalGeneration by training the model to only spit out one classification token per image.

Yeah I've attached an score layer at the top of the model by passing those last layer hidden states. While doing this, some cuda related issues ("data in different devices cuda:0 and cuda:1") came up.

So I've changed my LlavaForConditionalGeneration class itself in modelling_llava.py file (added score layer) and used that for classification.

Thankyou.

Zuhashaik commented 2 months ago

I am facing lots of data type and quantization errors while defining classifier (score) separately please initiate LlavaForSequenceClassification and SequenceClassification for vision models..

Hi @Zuhashaik, thanks for opening this issue!

We only add specific task heads for models if there are official checkpoints available; the task is described in the original paper or there's a large community demand.

Thank you!