search

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
6.35k stars 1.68k forks source link

Better Document Available Tasks #1698

Open Harryalways317 opened 4 months ago

Harryalways317 commented 4 months ago

we can include somewhere in the main readme,md saying we can get the list of tasks using the below command

lm-eval --tasks list
haileyschoelkopf commented 4 months ago

Would you be willing to open a PR adding a mention of this to the readme?

Listing the available tasks better (as well as brief descriptions) is on our todo list as well.

Harryalways317 commented 4 months ago

Yep,i'll add it to readme including task descriptions and make a PR, you can assign it to me

Thanks

haileyschoelkopf commented 4 months ago

I think for now just adding a mention of --tasks list is okay!

cnut1648 commented 4 months ago

would it be possible to also list the number of instances for each task?

Harryalways317 commented 4 months ago

Number of instances in the sense? Like Compute and memory requirements?

nairbv commented 4 months ago

I'm glad to see this exists. I've just been relying on the directory list to see what's supported (checking https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks).

Is there a more thorough list somewhere within the repo? I think it would also be nice to have a page somewhere with a table with the task's 'key' to use it, full name, relevant paper, and a description.

haileyschoelkopf commented 4 months ago

@nairbv we'll be adding that soon!

Harryalways317 commented 4 months ago

@haileyschoelkopf I added a page on tasks.md(https://github.com/Harryalways317/lm-evaluation-harness/blob/main/lm_eval/tasks/tasks.md) which has tasks family description and dataset url, all the urls are static, but ill try including paper and datasets urls there it self and make a pr

nairbv commented 4 months ago

@Harryalways317 that looks super helpful.

I think it'd be ideal if there was a way we could we could represent this info in one place (e.g. yaml or son) that could be imported into the markdown, and could also display the descriptions in the tasks --list output. Unfortunately I don't think there's any way to embed an external file into a markdown doc though, so maybe this just isn't possible? Maybe someone has another idea that would work.

I might suggest making the name in the "task family" column into a link, then we don't need a link column (or at least won't need two link columns if we want one for the paper and one for the dataset). Right now it looks like the links in your branch don't go anywhere.

Also we'll might want to update the "How to Contribute" section of the main README, adding a step to update the list here when a contributor adds a new task.

sepiatone commented 4 months ago

@nairbv @Harryalways317 How about adding this file (tasks.md #1727) with the table under docs and having two columns -"Task Family" and "Description". Each row of "Description" would be the description which would also be a link to the README.md for that specific task family under lm_eval/tasks/.

The specific READMEs for a task family would have all the information, including dataset links (like how it is now).

So when a contributor adds a new task, they'll add (1) a README for the task in the task folder and (2) another row to the table in this file in the docs folder

nairbv commented 4 months ago

that sounds good but I'd probably want the "task family" column to be the link. having the full desciption be a link could look awkward vs just having the one key be a link.

haileyschoelkopf commented 4 months ago

@Harryalways317 @nairbv @sepiatone this looks very helpful! It'd be a great contribution. +1 to having the family name be the link to the readme. Having a column that mentions either the main group names / task naming conventions / or just mentions whether the tasks are generative or not would also be ideal, though not strictly necessary to be an improvement over the current state of things.

EDIT: alternately, we can enforce that one should go to the task's README to view task naming conventions + group names.

I think it'd be ideal if there was a way we could we could represent this info in one place (e.g. yaml or son) that could be imported into the markdown, and could also display the descriptions in the tasks --list output. Unfortunately I don't think there's any way to embed an external file into a markdown doc though, so maybe this just isn't possible? Maybe someone has another idea that would work.

@lintangsutawika is looking into this soon (reporting an auto-generated markdown table via --tasks list, with each task having a 1-sentence description that gets reported in that table) in the meantime! I think that can complement this higher-level table though, this current proposed one is already quite useful.

haileyschoelkopf commented 3 months ago

A first iteration of this will land in #1901 . Feedback for further improvement welcome!

haileyschoelkopf commented 3 months ago

Thanks @Harryalways317 @anthony-dipofi and others who helped get a listing of each task's subfolder merged!

Leaving this issue open as a place to discuss continued improvements on this front.