search

Lightning-AI / litgpt

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.
https://lightning.ai
Apache License 2.0
6.85k stars 726 forks source link

Fix evaluation if device not specified #1344

Closed awaelchli closed 3 weeks ago

awaelchli commented 3 weeks ago

The command we have in the docs fails:

litgpt evaluate \
  --checkpoint_dir checkpoints/microsoft/phi-2/ \
  --batch_size 16 \
  --tasks "hellaswag" \
  --out_dir evaluate_model
Traceback (most recent call last):
  File "/home/zeus/miniconda3/envs/cloudspace/bin/litgpt", line 8, in <module>
    sys.exit(main())
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litgpt/__main__.py", line 143, in main
    fn(**kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litgpt/eval/evaluate.py", line 99, in convert_and_evaluate
    model = HFLM(repo_id, state_dict=state_dict, device=device, batch_size=batch_size, dtype=dtype)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lm_eval/models/huggingface.py", line 143, in __init__
    assert isinstance(device, str)
AssertionError

Because device=None is not accepted in lm_eval even though the type would suggest it is. This could be considered a bug, but we can choose a good default based on device availability.