search

hkust-nlp / ceval

Official github repo for C-Eval, a Chinese evaluation suite for foundation models [NeurIPS 2023]
https://cevalbenchmark.com/
MIT License
1.63k stars 78 forks source link

prompt大于max_len时的处理方式？ #24

Closed bbyjlb closed 1 year ago

bbyjlb commented 1 year ago

在验证llama效果 请教一下5-shot情况下，部分subject的prompt的token数大于2048的情况下，是怎么处理的呢？看llama.py的代码里貌似没有相关的逻辑且会报错

复现： torchrun --nproc_per_node 1 code/evaluator_series/eval_llama.py --ckpt_dir [ckpd dir] --param_size 7 --few_shot --ntrain 5 --subject logic

File "~/ceval/code/evaluator_series/evaluators/llama.py", line 109, in generate tokens[0, : prompt_size] = torch.tensor(prompt_tokens).long() RuntimeError: The expanded size of the tensor (2048) must match the existing size (2318) at non-singleton dimension 0. Target sizes: [2048]. Tensor sizes: [2318]

jxhe commented 1 year ago

这种情况在非cot测试的setting下应该几乎没有。具体出现的时候我们是会dynamically减少few-shot的数量来fit context window