search

state-spaces / mamba

Mamba SSM architecture
Apache License 2.0
13.23k stars 1.13k forks source link

lm_harness_eval evaluation error while running #299

Open lijiek opened 7 months ago

lijiek commented 7 months ago

when I do evaluation for lm_harness_eval, there are some error as below. what is problem in it? and how to deal with it to make it work successfully? Thanks so much.

python evals/lm_harness_eval.py --model mamba --model_args pretrained=state-spaces/mamba-130m --tasks lambada_openai,hellaswag,piqa,arc_easy,arc_challenge,winogrande --device cuda --batch_size 64 2024-04-19:15:48:12,682 INFO [utils.py:148] Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2024-04-19:15:48:12,682 INFO [utils.py:160] NumExpr defaulting to 8 threads. 2024-04-19:15:48:12,932 INFO [config.py:58] PyTorch version 2.2.2 available. 2024-04-19:15:48:17,024 WARNING [templates.py:384] Tried instantiating DatasetTemplates for gsmk boolq, but no prompts found. Please ignore this warning if you are creating new prompts for this dataset. 2024-04-19:15:48:17,025 WARNING [templates.py:384] Tried instantiating DatasetTemplates for EleutherAI/asdiv, but no prompts found. Please ignore this warning if you are creating new prompts for this dataset. 2024-04-19:15:48:17,804 INFO [main.py:184] Selected Tasks: ['arc_challenge', 'arc_easy', 'hellaswag', 'lambada_openai', 'piqa', 'winogrande'] Traceback (most recent call last): File "evals/lm_harness_eval.py", line 39, in cli_evaluate() File "/home/ubuntu/PointCloudProjects/mamba/3rdparty/lm-evaluation-harness/lm_eval/main.py", line 186, in cli_evaluate results = evaluator.simple_evaluate( File "/home/ubuntu/PointCloudProjects/mamba/3rdparty/lm-evaluation-harness/lm_eval/utils.py", line 343, in _wrapper return fn(*args, kwargs) File "/home/ubuntu/PointCloudProjects/mamba/3rdparty/lm-evaluation-harness/lm_eval/evaluator.py", line 90, in simple_evaluate lm = lm_eval.api.registry.get_model(model).create_from_arg_string( File "/home/ubuntu/PointCloudProjects/mamba/3rdparty/lm-evaluation-harness/lm_eval/api/model.py", line 140, in create_from_arg_string return cls(args, args2) File "evals/lm_harness_eval.py", line 22, in init self._model = MambaLMHeadModel.from_pretrained(pretrained, device=device, dtype=dtype) File "/home/ubuntu/anaconda3/envs/mamba/lib/python3.8/site-packages/mamba_ssm/models/mixer_seq_simple.py", line 244, in from_pretrained config = MambaConfig(config_data) TypeError: init() got an unexpected keyword argument 'architectures'

tridao commented 5 months ago

I've changed the instruction in the README to use lm-eval from PyPI instead our own submodule, hopefully that fixes this issue.