search

turboderp / exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs
MIT License
3.53k stars 274 forks source link

[BUG] test_mmlu.py doesn't produce any results #280

Closed ThomasBaruzier closed 8 months ago

ThomasBaruzier commented 8 months ago

Example:

(env) ~/files/ai/exllama/tests python test_mmlu.py 
 -- Loading dataset: cais/mmlu/anatomy...
/home/tyra/files/ai/exllama/env/lib/python3.10/site-packages/datasets/load.py:1429: FutureWarning: The repository for cais/mmlu contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/cais/mmlu
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
 -- Loading dataset: cais/mmlu/computer_security...
 -- Loading dataset: cais/mmlu/formal_logic...
 -- Loading dataset: cais/mmlu/logical_fallacies...
 -- Loading dataset: cais/mmlu/philosophy...
 -- Loading dataset: cais/mmlu/nutrition...
 -- Finished

;anatomy;computer_security;formal_logic;logical_fallacies;philosophy;nutrition

(env) ~/files/ai/exllama/tests

I haven't tested with other benchmarks. Perplexity evaluation is working with test_inference.py, though.

Environment: Arch Linux RTX 3090 CUDA 12.1.105 Exllama V2 commit 7d37b50d90908f899a45ef8ee901a4105c19fbe3

Is there anything I can do get this script to work? Thanks.

turboderp commented 8 months ago

It looks like the variants list is empty? The script is written to test multiple versions of the same model. You'd point it to /mnt/models/my_model and it would scan for /mnt/models/my_model/3.0bpw, /mnt/models/my_model/4.0bpw etc. For a quick fix you should be able to just do:

model_base = "/mnt/models"
variants = ["my_model"]

Or something like that.

ThomasBaruzier commented 8 months ago

Yes, that was the issue. Thank you very much!