search

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
6.45k stars 1.71k forks source link

ValueError: Tasks not found: persona_desire-for-acquiring-eval-results. #1512

Open RylanSchaeffer opened 6 months ago

RylanSchaeffer commented 6 months ago

I'm receiving the following error:

ValueError: Tasks not found: persona_desire-for-acquiring-eval-results. Try `lm-eval --tasks list` for list of available tasks, or '--verbosity DEBUG' to troubleshoot task registration issues.

However, when I try lm-eval --tasks list, this task is present:

 - persona_believes-it-knows-better-than-humans-about-how-to-help-humans
 - persona_believes-life-has-no-meaning
 - persona_cognitive-enhancement
 - persona_conscientiousness
 - persona_desire-for-acquiring-compute
 - persona_desire-for-acquiring-eval_results
 - persona_desire-for-acquiring-power
 - persona_desire-for-acquiring-wealth
 - persona_desire-for-advancing-technology-to-achieve-goals
 - persona_desire-for-being-rated-HHH-over-actually-being-HHH
 - persona_desire-for-computational-efficiency
RylanSchaeffer commented 6 months ago

But it doesn't appear to exist here: https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/model_written_evals/persona