Fine-tuning MPNet model

[ahdmmtq]

I decided to use paraphrase-mpnet-base-v2 word embedding for a regression task.

I tried fine-tuning the model via adding a layer on top of it and training it so it should receive a single sentence at time and predict the score of this sentence.

This what I tried so far:

from sentence_transformers import SentenceTransformer, InputExample, losses, models
from torch.utils.data import DataLoader
from torch import nn

word_embedding_model = SentenceTransformer('paraphrase-mpnet-base-v2')
dense_model = models.Dense(in_features=768, out_features=768, activation_function=nn.Linear(in_features=768, out_features=1))

model = SentenceTransformer(modules=[word_embedding_model, dense_model])

#Define your train examples. You need more than just two examples...
train_examples = InputExample(texts=['My first sentence', 'My second sentence', '3rd sentence'], label=[0.8, 1.5, 3]),

#Define your train dataset, the dataloader and the train loss
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=16)
train_loss = losses.MSELoss(model)

#Tune the model
model.fit(train_objectives=[(train_dataloader, train_loss)], epochs=1, warmup_steps=100)

However there is something wrong with my model and it never gets better when trained, it may also get worse and worse.

I don't know what's wrong, may be the input layer? I just discovered this library 2 days ago and I'm reading the docs since then but I still don't know how to get this task done.

[nreimers]

SentenceTransformer is the wrong class, you should use the CrossEncoder class: https://www.sbert.net/examples/applications/cross-encoder/README.html https://www.sbert.net/examples/training/cross-encoder/README.html

Instead of sentence pairs with a score you can pass also individual sentences with a pair.