search

aryopg / mmlu-redux

Creative Commons Attribution 4.0 International
7 stars 0 forks source link

about config #2

Open Violettttee opened 2 months ago

Violettttee commented 2 months ago

is the code missing the config?

Clipboard_Screenshot_1726735533

and i see two datasets from the code and paper one is : dataset = load_dataset( "edinburgh-dawg/labelchaos", args.config, split="test", token=HF_READ_TOKEN )

another is the dataset mentioned in this repositories.

Clipboard_Screenshot_1726735923

whats the difference?

saxenarohit commented 1 month ago

Hi, Thanks for using MMLU-Redux!

The config refers to the subset of the dataset. For labelchaos, you can select any one of these ['bad_options_clarity', 'bad_questions_clarity', 'clean', 'clean_subsampled', 'multiple_correct_answers', 'no_correct_answer', 'small', 'wrong_groundtruth']

For information on dataset differences, please refer to the readme under Supervised Fine-tuning--> LabelChaos, where we have explained the use of the LabelChaos dataset. https://github.com/aryopg/mmlu-redux?tab=readme-ov-file#labelchaos

Thanks.