Open Harryalways317 opened 4 months ago
Would you be willing to open a PR adding a mention of this to the readme?
Listing the available tasks better (as well as brief descriptions) is on our todo list as well.
Yep,i'll add it to readme including task descriptions and make a PR, you can assign it to me
Thanks
I think for now just adding a mention of --tasks list
is okay!
would it be possible to also list the number of instances for each task?
Number of instances in the sense? Like Compute and memory requirements?
I'm glad to see this exists. I've just been relying on the directory list to see what's supported (checking https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks).
Is there a more thorough list somewhere within the repo? I think it would also be nice to have a page somewhere with a table with the task's 'key' to use it, full name, relevant paper, and a description.
@nairbv we'll be adding that soon!
@haileyschoelkopf I added a page on tasks.md(https://github.com/Harryalways317/lm-evaluation-harness/blob/main/lm_eval/tasks/tasks.md) which has tasks family description and dataset url, all the urls are static, but ill try including paper and datasets urls there it self and make a pr
@Harryalways317 that looks super helpful.
I think it'd be ideal if there was a way we could we could represent this info in one place (e.g. yaml or son) that could be imported into the markdown, and could also display the descriptions in the tasks --list
output. Unfortunately I don't think there's any way to embed an external file into a markdown doc though, so maybe this just isn't possible? Maybe someone has another idea that would work.
I might suggest making the name in the "task family" column into a link, then we don't need a link column (or at least won't need two link columns if we want one for the paper and one for the dataset). Right now it looks like the links in your branch don't go anywhere.
Also we'll might want to update the "How to Contribute" section of the main README, adding a step to update the list here when a contributor adds a new task.
@nairbv @Harryalways317 How about adding this file (tasks.md #1727) with the table under docs and having two columns -"Task Family" and "Description". Each row of "Description" would be the description which would also be a link to the README.md for that specific task family under lm_eval/tasks/
The specific READMEs for a task family would have all the information, including dataset links (like how it is now).
So when a contributor adds a new task, they'll add (1) a README for the task in the task folder and (2) another row to the table in this file in the docs folder
that sounds good but I'd probably want the "task family" column to be the link. having the full desciption be a link could look awkward vs just having the one key be a link.
@Harryalways317 @nairbv @sepiatone this looks very helpful! It'd be a great contribution. +1 to having the family name be the link to the readme. Having a column that mentions either the main group names / task naming conventions / or just mentions whether the tasks are generative or not would also be ideal, though not strictly necessary to be an improvement over the current state of things.
EDIT: alternately, we can enforce that one should go to the task's README to view task naming conventions + group names.
I think it'd be ideal if there was a way we could we could represent this info in one place (e.g. yaml or son) that could be imported into the markdown, and could also display the descriptions in the tasks --list output. Unfortunately I don't think there's any way to embed an external file into a markdown doc though, so maybe this just isn't possible? Maybe someone has another idea that would work.
@lintangsutawika is looking into this soon (reporting an auto-generated markdown table via --tasks list
, with each task having a 1-sentence description that gets reported in that table) in the meantime! I think that can complement this higher-level table though, this current proposed one is already quite useful.
A first iteration of this will land in #1901 . Feedback for further improvement welcome!
Thanks @Harryalways317 @anthony-dipofi and others who helped get a listing of each task's subfolder merged!
Leaving this issue open as a place to discuss continued improvements on this front.
we can include somewhere in the main readme,md saying we can get the list of tasks using the below command