search

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
6.91k stars 1.84k forks source link

mgsm tasks not found when using Accelerate #2405

Open Mugariya opened 3 weeks ago

Mugariya commented 3 weeks ago

If I try to run mgsm task using accelerate, I have the tasks not found error. But if I run it outside accelerate, it works fine.

Any idea how I can incorporate all tasks with Accelerate?

This is the command I was using: accelerate launch --multi_gpu --num_processes 4 -m lm_eval model: hf model_args: pretrained=${model} tasks: mgsm_direct device: cuda batch_size: auto

baberabb commented 2 weeks ago

Hi! not sure what the underlying cause could be. Can you run it with --verbosity DEBUG and see if any mgsm specific warning/error shows up. I would also double check the environments.