self.compute_metrics in the kmoe_trainer

[YerongLi]

Where is the self.compute_metrics in the kmoe_trainer.py?

        if self.compute_metrics is not None and preds is not None and label_ids is not None:
            metrics = self.compute_metrics(EvalPrediction(predictions=preds, label_ids=label_ids))
        else:
            metrics = {}

[wyu97]

Line 294. https://github.com/DM2-ND/MoKGE/blob/b36a6908c62a7fc5d5ae33e832d28d0cb3c5ffbb/trainers/kgmoe_trainer.py#L294

[YerongLi]

Thanks for instant reply. With your example script for eg dataset: scripts/KGMixtureOfExpertCho.sh,

self.compute_metrics is None all the time right? For instance, in the script you pointed out, I still cannnot figure out how you assign self.compute_metrics. Can you show us how to compute the metric?

I am just wondering on your task for eg, how you compute the scores with 3 reference sentences.

Because preds and label_ids are both of size (B x 60)

[wyu97]

Good question!

I did modify a little bit since the self.compute_metric is predefined by Huggingface Seq2Seq models. Since the prediction size is different from the input size, I add the evaluation script in

https://github.com/DM2-ND/MoKGE/blob/b36a6908c62a7fc5d5ae33e832d28d0cb3c5ffbb/trainers/seq2seq_trainer.py#L338

and did not use the self.compute_metrics

[YerongLi]

I see, it's in the training loop, not the prediction loop.