Using Callback class for metrics like F1, Recall and Precision with SetFitTrainer

Hi there,

I am very new to using transformer models space and especially to setfit. My task to build a classification model using Setfit (Few shot text classification) as I just have 500 customer reviews to tag into various categories (18 t0 be precise). My questions is that once we sample the dataset to N samples per M classes (N*M), in that scenario it will be fine to use "Accuracy" as the metric but what if I don't sample my classes to be balanced but just pass on the most important customer reviews per class ( could be 10 reviews for class1 and 20 for class2 ) to the setfit trainer, in that scenario if I am not wrong I need to measure the F1, recall and precision, correct? And if that's the case, then I understand that we need to create a callback to pass such metric, but when I am doing that using the following code. I am getting an error: " NameError: name 'TrainerCallback' is not defined "

class CustomCallback(TrainerCallback):

    def __init__(self, trainer) -> None:
        super().__init__()
        self._trainer = trainer

    def on_epoch_end(self, args, state, control, **kwargs):
        if control.should_evaluate:
            control_copy = deepcopy(control)
            self._trainer.evaluate(eval_dataset=self._trainer.train_dataset)
            return control_copy

def compute_metrics(pred): global num_labels labels = pred.labelids preds = pred.predictions.argmax(-1) precision, recall, f1, = precision_recall_fscore_support(labels, preds, average='macro') acc = accuracy_score(labels, preds) loss_fct = CrossEntropyLoss() logits = torch.tensor(pred.predictions) labels = torch.tensor(labels) loss = loss_fct(logits.view(-1, num_labels), labels.view(-1)) return { 'f1': f1, 'precision': precision, 'recall': recall }

training_args = TrainingArguments( num_train_epochs=1, loss_class=CosineSimilarityLoss, batch_size=10, num_iterations=5, # The number of text pairs to generate for contrastive learning

trainer = SetFitTrainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=test_dataset, compute_metrics=compute_metrics )

trainer.add_callback(CustomCallback(trainer))

train_result = trainer.train()

metrics = trainer.evaluate()

Is my approach correct? If yes, then how can I resolve this issue? And how do I print all these different metric scores?

huggingface / setfit

Using Callback class for metrics like F1, Recall and Precision with SetFitTrainer #255