Llama2-7B-chat-4k测试出来结果不一样

[PengWenChen]

Reopen issue #55 嗨 @bys0318 (@slatter666) 您好~ 我嘗試跑 Llama2-7B-chat-4k(https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) 出來的結果也跟您 leaderboard 上的成績不同，差異蠻大的。 我的執行環境不能連網，因此跟您的 pred.py 唯一的差異是 loading data & model & tokenizer from local 而已。 想請問成績為何有如此大的差異呢? 謝謝 我跑出來的成績 (seed 使用 pred.py 原始的 seed-42): { "narrativeqa": 14.57, "qasper": 6.6, "multifieldqa_en": 3.65, "multifieldqa_zh": 4.29, "hotpotqa": 4.27, "2wikimqa": 5.67, "musique": 1.3, "dureader": 15.71, "gov_report": 24.53, "qmsum": 16.13, "multi_news": 2.41, "vcsum": 0.03, "trec": 68.0, "triviaqa": 88.59, "samsum": 41.38, "lsht": 19.75, "passage_count": 0.5, "passage_retrieval_en": 3.0, "passage_retrieval_zh": 0.0, "lcc": 66.64, "repobench-p": 60.06 }

Copy results from github leaderboard: { "narrativeqa": 18.7, "qasper": 19.2, "multifieldqa_en": 36.8, "multifieldqa_zh": 11.9, "hotpotqa": 25.4, "2wikimqa": 32.8, "musique": 9.4, "dureader": 5.2, "gov_report": 27.3, "qmsum": 20.8, "multi_news": 25.8, "vcsum": 0.2, "trec": 61.5, "triviaqa": 77.8, "samsum": 40.7, "lsht": 19.8, "passage_count": 2.1, "passage_retrieval_en": 9.8, "passage_retrieval_zh": 0.5, "lcc": 52.4, "repobench-p": 43.8 }

我的資料是下載您README中提供的link:https://huggingface.co/datasets/THUDM/LongBench/resolve/main/data.zip load data by data = [json.loads(line for line in open(path, "r", encoding="utf-8")] load tokenizer model by tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, torch_dtyp=torch.bfloat16).to(device)

[BlackieMia]

我也复现了类似的结果I got the similar result of yours.

[PengWenChen]

Sorry I just found out I accidentally use llama2-7B model instead of llama2-7B-chat model. The chat version model's scores I run is: { "narrativeqa": 18.82, "qasper": 23.65, "multifieldqa_en": 36.52, "multifieldqa_zh": 10.59, "hotpotqa": 26.4, "2wikimqa": 31.85, "musique": 7.76, "dureader": 5.2, "gov_report": 26.56, "qmsum": 21.28, "multi_news": 26.3, "vcsum": 0.18, "trec": 65.0, "triviaqa": 83.17, "samsum": 41.0, "lsht": 18.75, "passage_count": 1.57, "passage_retrieval_en": 7.5, "passage_retrieval_zh": 9.5, "lcc": 59.04, "repobench-p": 52.91 } I think it's pretty closed to the numbers of those on the leaderboard.