Low MMLU of llama2

[Zkli-hub]

Reminder

• [X] I have read the README and searched the existing issues.

System Info

• llamafactory version: 0.8.3.dev0
• Platform: Linux-6.5.0-18-generic-x86_64-with-glibc2.35
• Python version: 3.10.12
• PyTorch version: 2.3.1+cu121 (GPU)
• Transformers version: 4.41.2
• Datasets version: 2.20.0
• Accelerate version: 0.31.0
• PEFT version: 0.11.1
• TRL version: 0.9.4
• GPU type: NVIDIA A800 80GB PCIe
• Bitsandbytes version: 0.43.1

Reproduction

CUDA_VISIBLE_DEVICES=0 llamafactory-cli eval --model_name_or_path PATH --template fewshot --task mmlu --split test --lang en --n_shot 5 --batch_size 1

Expected behavior

Evaluate the MMLU performance of llama2-7b-base, but the MMLU results is lower than the original paper which is 45.3

Others

No response

[hiyouga]

A small derivation is acceptable and usual