Closed veekaybee closed 7 months ago
When you run Ray evaluation job on a quantized model, it requires GPUs, per the bitsandbytes quantization environment validation checker.
We can make this clearer in the documentation for example evaluation runs.
Check documenation
Addressing this issue: https://github.com/mozilla-ai/lm-buddy/issues/50
Also added a config file for a quantized HF model eval run.
What's changing
When you run Ray evaluation job on a quantized model, it requires GPUs, per the bitsandbytes quantization environment validation checker.
We can make this clearer in the documentation for example evaluation runs.
How to test it
Check documenation
Related Jira Ticket
Addressing this issue: https://github.com/mozilla-ai/lm-buddy/issues/50
Additional notes for reviewers
Also added a config file for a quantized HF model eval run.