launch lighteval using `lighteval --args` - Githubissues

huggingface / lighteval

LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.

MIT License

467 stars 54 forks source link

launch lighteval using `lighteval --args` #152

Open NathanHB opened 2 months ago

NathanHB commented 2 months ago

removes uneeded script files to launch lighteval
moves parsers function to lighteval.commands.parsers
creates a lighteval cli executable with 3 subcommands (list-tasks, nanotron, accelerate)

how to use:

with nanotron

nothing changes except for the use of the cli executable

 lighteval nanotron  {nanotron args}

with accelerate

accelerate args now need to be passed to the lighteval script.

accelerate launch --multi_gpu --num_processes=8 -m \
lighteval accelerate \
--model_args "pretrained=gpt2,dtype=float16" \
--tasks "original|mmlu:anatomy|0|0" \
--override_batch_size 1  \
--output_dir="tmp/" \
--max_samples 1 \
--use_chat_template