Using Finbert for 240 multilabel multiclass classification

[pratikchhapolika]

I have label of dimension 240, it is multi label classification problem.

I downloaded the Finbert model:

from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer
from transformers import AutoTokenizer

tokenizer=AutoTokenizer.from_pretrained("/home/pratik/finbert", use_fast=True)

train_tokenizer_texts = list(map(lambda t: tokenizer.tokenize(t,add_special_tokens=True,max_length=512,padding='max_length'), tqdm(train_sentences)))

np.array(train_tokenizer_texts[0])

%%time
#Inititaing a BERT model
model = AutoModelForSequenceClassification.from_pretrained("/home/pratik/finbert", num_labels = 240)
model.cuda()

This gives me error:

RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
    size mismatch for classifier.weight: copying a param with shape torch.Size([3, 768]) from checkpoint, the shape in current model is torch.Size([240, 768]).
    size mismatch for classifier.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([240]).

[doguaraci]

The model weights include a classifier layer with three labels. That is why you get an error. You need to replace that last layer.

You should probably do something like the following:

finbert_without_classifier_head = AutoModel.from_pretrained("/home/pratik/finbert")
finbert_without_classifier_head.save_pretrained("/home/pratik/finbert_wch")
finbert_twoforty = AutoModelForSequenceClassification.from_pretrained("/home/pratik/finbert_wch", num_labels = 240)