search

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
6.39k stars 1.69k forks source link

chatglm2 acc=0 on lambada_openai dataset, is it correct? #959

Open changwangss opened 10 months ago

changwangss commented 10 months ago

I also validate chatglm and chatglm3, they are works, could you have fix the root cause? https://huggingface.co/THUDM/chatglm2-6b/discussions/97 python main.py --model hf-causal --model_args pretrained=THUDM/chatglm2-6b,trust_remote_code=True --tasks lambada_openai --limit 10 --batch_size 1 --no_cache

StellaAthena commented 10 months ago

When I run this I get AttributeError: property 'pad_token_id' of 'ChatGLMTokenizer' object has no setter. You're seeing it run though?

changwangss commented 10 months ago

as the code show, https://huggingface.co/THUDM/chatglm2-6b/blob/main/tokenization_chatglm.py#L91 yes, correct, the pad_token or pad_token_id is property and has no setter. I comment tokenizer.pad_token = tokenizer.eos_token, I can get the acc about chatglm&chatglm3, but acc=0 for chatglm2. Could you give me some help?

haileyschoelkopf commented 8 months ago

ChatGLM-6B and ChatGLM3-6b both seem to work fine. I'm still unsure as to what's going wrong on ChatGLM2 and would very much like to figure out how to fix it.