Open shubhamagarwal92 opened 10 months ago
Hi @philschmid!
If I would like to compute metrics while training (similar to https://www.philschmid.de/fine-tune-flan-t5), how should I change the compute metrics function to pass to SFT trainer for Llamav2 (https://www.philschmid.de/instruction-tune-llama-2)
Would this be same as:
def compute_metrics(eval_pred, tokenizer): predictions, labels = eval_pred decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True) labels = np.where(labels != -100, labels, tokenizer.pad_token_id) decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True) result = rouge.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True) prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in predictions] result["gen_len"] = np.mean(prediction_lens) return {k: round(v, 4) for k, v in result.items()}
Specifically, do I still need to do
labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
Is there a way to save the predictions decoded_preds while also passing the filename=f"{savedir}/{preds}.csv" to Trainer?
decoded_preds
filename=f"{savedir}/{preds}.csv"
You can write a callback for the Trainer which is executed after an evaluation phase. https://huggingface.co/docs/transformers/main_classes/callback#transformers.TrainerCallback.on_evaluate
Trainer
Hi @philschmid!
If I would like to compute metrics while training (similar to https://www.philschmid.de/fine-tune-flan-t5), how should I change the compute metrics function to pass to SFT trainer for Llamav2 (https://www.philschmid.de/instruction-tune-llama-2)
Would this be same as:
Specifically, do I still need to do
labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
Is there a way to save the predictions
decoded_preds
while also passing thefilename=f"{savedir}/{preds}.csv"
to Trainer?