search

hiyouga / LLaMA-Factory

Unified Efficient Fine-Tuning of 100+ LLMs (ACL 2024)
https://arxiv.org/abs/2403.13372
Apache License 2.0
33.82k stars 4.16k forks source link

sft在eval或者predict阶段输出每条样本的log_prob_sum #5950

Open sc89703312 opened 3 hours ago

sc89703312 commented 3 hours ago

Reminder

System Info

现在sft的do_eval阶段只能输出整体的eval loss，do_predict阶段只能输出prediction response；能否支持下输出给定<prompt, response>的log_prob_sum等，方便做一些离线分析

Reproduction

python3 src/train.py \ --stage sft \ --model_name_or_path /mnt/bn/shengchen-sft-yg/finetune_output/$MODEL/checkpoint-$step \ --do_predict \ --eval_dataset $dateset \ --dataset_dir data/ \ --template default \ --finetuning_type full \ --output_dir /mnt/bn/shengchen-sft-yg/finetune_output/$MODEL/evaluation-$step \ --per_device_eval_batch_size 8 \ --max_samples 1000000 \ --predict_with_generate \ --bf16 \ --repetition_penalty 1.0

Expected behavior

No response

Others

No response

sc89703312 commented 2 hours ago

和 https://github.com/hiyouga/LLaMA-Factory/issues/1447 这个issue的解法有点类似，能否做成一个标准化的选项