Unable to reproduce PPL for GPT-Neo-125M using lm-eval

Langboat / mengzi-retrieval-lm

An experimental implementation of the retrieval-enhanced language model

Apache License 2.0

75 stars 5 forks source link

Unable to reproduce PPL for GPT-Neo-125M using lm-eval #16

Open pedrogengo opened 1 year ago

pedrogengo commented 1 year ago

Hey!

I'm trying to run the following command using the lm-eval cli, but I can't reproduce the results you shared. Did you do something different? If not, do you have any idea where I'm doing wrong?

python main.py \
    --model gpt2 \
    --model_args pretrained=EleutherAI/gpt-neo-125M \
    --device 0 \
    --tasks wikitext \
    --batch_size 1

bling0830 commented 1 year ago

May I ask what is the result of the EleutherAI/gpt-neo-125? Did you get an error when using it?