microsoft / LMOps

General technology for enabling AI capabilities w/ LLMs and MLLMs
https://aka.ms/GeneralAI
MIT License
3.7k stars 283 forks source link

[UPRISE]What is the use of some keys in hellaswag and copa? #282

Open zhouchang123 opened 5 days ago

zhouchang123 commented 5 days ago

What is the use of some keys in hellaswag and copa? For example: copa:{ "premise": "The woman filed a restraining order against the man.", "choice1": "The man called her.", "choice2": "The man stalked her.", "question": "cause", "idx": 302, "label": 1, "id": 0, "task_name": "copa" } What's the use of “idx” in copa? hellaswag:{ "ind": 34409, "activity_label": "Home and Garden", "ctx_a": "[header] How to make a butterfly out of plastic spoons [title] Gather the materials you will need for this project, listed below. [title] Put a craft cloth or some newspaper down on your working surface. [title] Cut the top portion of the four spoons off (leaving about half an inch of the handle left.", "ctx_b": "", "ctx": "[header] How to make a butterfly out of plastic spoons [title] Gather the materials you will need for this project, listed below. [title] Put a craft cloth or some newspaper down on your working surface. [title] Cut the top portion of the four spoons off (leaving about half an inch of the handle left.", "endings": [ "[step] ) the top portion is the flat spoon part. [title] Using the cut-off handle of one of the spoons, make that the \" body \" of the butterfly.", "[step] As the buttons stretch they will process the solution. ) [title] Put one of the red plastic buttons (left over for the butterfly) on top of the red plastic spoon.", "[step] And when done, you should have 8 small, square, circular spoons. [title] Use these four spoons as either a mirror, or a butterfly.", "[step] ) [title] Thread a thin needle into a cast iron ring with four ends attached. [step] Thread the needle to the ring, and then sew the top portion onto the handle of the spoon at the top, or the side which should be facing away from the spoon." ], "source_id": "wikihow~56680", "split": "train", "split_type": "indomain", "label": "0", "id": 0, "task_name": "hellaswag" } What's the use of "ind",“activity_label”,"ctx_a","ctx_b","source_id","split","split_type" in copa?

zhouchang123 commented 5 days ago

Could you please explain the quest@cdxeve

cdxeve commented 18 hours ago

Hi, I've included code in task.py for processing the relevant information in each dataset example. Any additional information (such as those keys you've listed above) that isn't needed is simply ignored during processing.