Curious, why remove `mmlu` and `drop` from the list of datasets?

jondurbin / bagel

A bagel, with everything.

300 stars 31 forks source link

Curious, why remove `mmlu` and `drop` from the list of datasets? #5

Closed vgoklani closed 5 months ago

vgoklani commented 5 months ago

Hey there,

Curious, why did you remove mmlu and drop?

Are they considered low-quality datasets for training? Or do you want to avoid any potential leakage with the evaluation frameworks?

Thanks!

jon-convai commented 5 months ago

MMLU doesn't really seem to add much to the model's abilities, and most people don't use LLMs for multiple choice anyway, so it seems like it's just wasted compute cycles.

DROP is somewhat more helpful, for RAG, but the answers are extremely short. I also include ROPES, airoboros context-obedient items, and squadv2 train, so this particular dataset is a bit redundant.