--no_cache was always needed. Without this option the code would fail due to incompatibility in lm-eval-harness.
In lm_eval/evaluatesimple_evaluate() accepts model or string with model name. However here in the if branch without no_cache the code adds model to string. Since always pass loaded model there - adding it to string causes error. This is an issue in the older lm-eval harness code that we copied here. So I hardcoded it to be always True.
--no_cache was always needed. Without this option the code would fail due to incompatibility in lm-eval-harness. In
lm_eval/evaluatesimple_evaluate()accepts model or string with model name. However here in the if branch without no_cache the code adds model to string. Since always pass loaded model there - adding it to string causes error. This is an issue in the older lm-eval harness code that we copied here. So I hardcoded it to be always True.In the latest LM-eval code this error is already fixed with my commit.