search

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
6.74k stars 1.79k forks source link

Error: BuilderConfig not found #1444

Closed gopikrishnajha closed 8 months ago

gopikrishnajha commented 8 months ago

I am trying to run the following command.

lm_eval --model hf \
    --model_args pretrained=mistralai/Mistral-7B-v0.1 \
    --tasks pile_openwebtext2 \
    --device cpu \
    --batch_size 10

And getting this error: ValueError: BuilderConfig 'pile_openwebtext2' not found. Available: ['all', 'enron_emails', 'europarl', 'free_law', 'hacker_news', 'nih_exporter', 'pubmed', 'pubmed_central', 'ubuntu_irc', 'uspto', 'github']

I have checked lm-eval --tasks list and pile_openwebtext2 is available there.

haileyschoelkopf commented 8 months ago

Hi! This is a known issue ; I will push an update to the Pile tasks so that they reflect the fix described in #731 , but you will also need your own copy of the Pile downloaded locally to run these tasks.

closing as duplicative of #1338 for tracking purposes. please let us know if there are lingering questions though!