assert isinstance(pretrained, str)

[majacinka]

Hi!

First of all, I want to thank everyone involved in this great project!

I have a specific problem that I can't solve for hours, and I don't have a lot of programming experience, chatgpt and all the other chatbots can't help so I'm going to try here.

I'm trying to evaluate my model but I keep running into: "Error: File does not exist". This model in particular was already evaluated through open LLM leaderboard without a problem. I can also do inference. I have already checked "debug" and this is what I get in the runpod logs:

File "/lm-evaluation-harness/main.py", line 89, in <module>
main()
File "/lm-evaluation-harness/main.py", line 57, in main
results = evaluator.simple_evaluate(
File "/lm-evaluation-harness/lm_eval/utils.py", line 242, in _wrapper
return fn(*args, **kwargs)
File "/lm-evaluation-harness/lm_eval/evaluator.py", line 69, in simple_evaluate
lm = lm_eval.models.get_model(model).create_from_arg_string(
File "/lm-evaluation-harness/lm_eval/base.py", line 115, in create_from_arg_string
return cls(**args, **args2)
File "/lm-evaluation-harness/lm_eval/models/gpt2.py", line 36, in __init__
assert isinstance(pretrained, str)
AssertionError

[mlabonne]

Does it only say that for MMLU or for every benchmark?

[rhys101]

@majacinka You need to make sure that you're using the very latest Colab notebook (linked from the README in this repo).

To fix an older Colab notebook that you may be using, towards the end change the MODEL to MODEL_ID

from

    env={
        "BENCHMARK": BENCHMARK,
        "MODEL": MODEL,

to

    env={
        "BENCHMARK": BENCHMARK,
        "MODEL_ID": MODEL,