search

instructlab / eval

Python library for Evaluation
Apache License 2.0
5 stars 15 forks source link

mmlu isn't consuming multiple gpus #68

Open danmcp opened 1 month ago

danmcp commented 1 month ago

Steps to recreate:

Launch mmlu on an instance with multiple gpus. Run:

ilab model evaluate --model models/instructlab/granite-7b-lab --benchmark mmlu

Only 1 gpu is consumed. Adjusting batch-size doesn't seem to have any effect.

danmcp commented 1 month ago

This might be best fixed with https://github.com/instructlab/eval/issues/50