Fix extended tasks usage in README

huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

MIT License

690 stars 78 forks source link

Fix extended tasks usage in README #108

Closed lewtun closed 6 months ago

lewtun commented 6 months ago

The instructions to run ifeval are outdated in the README, so here's the proposed fix :)

Are custom_tasks now entirely deprecated? If so, it might be good to remove them also from run_evals_accelerate.py and other parts of the codebase

clefourrier commented 6 months ago

Hi! There is a difference between:

a core task (needs to specific processing, added by maintainers)
an extended task (unusual processing, added by maintainers)
a community task (added by the community)
a custom task (your specific current and local experiment)

When launching lighteval, all core tasks are available, extended tasks are all loaded from a specific flag, and community/custom tasks are loaded if specifically pointed to using an other specific flag.

lewtun commented 6 months ago

Thanks for the clarification! This would make IFEval an "extended" task right? If so, let me know if the PR makes sense or if I should close it or rephrase it :)

clefourrier commented 6 months ago

Ifeval is indeed extended :) If you could add any comment in the README which would make the above clearer, it would be super cool

lewtun commented 6 months ago

Ifeval is indeed extended :) If you could add any comment in the README which would make the above clearer, it would be super cool

Took a stab at explaining this here: https://github.com/huggingface/lighteval/pull/108/commits/a725d931a10e12dfc90552d32445efddf9d5b338