search

microsoft / promptbase

All things prompt engineering
MIT License
5.43k stars 302 forks source link

Enable 'all' as an MMLU subject #43

Closed riedgar-ms closed 10 months ago

riedgar-ms commented 10 months ago

Make it possible to run promptbase on the entire MMLU dataset via an all subject. This takes a simple "disk space is cheap" approach.

riedgar-ms commented 10 months ago

@Harsha-Nori @Nking92 ?

nking-1 commented 10 months ago

I'm ok with this for now, but in the future I would prefer if we just put everything into 1 file (all_questions.json). The JSON data for each question should have split and subject properties. That way we will always have only 1 file we look for, and we filter based on the split property if we want a specific split.

The way this is done now, there's an implicit dependency on having a filename formatted like mmlu_{subject}_{split}.json. In my opinion, subject and split are both data properties and should coincide with the question data itself, not the filename.

It seems like the MMLU testing code already has some assumptions about the filenames, so we would need to change that behavior in addition to this format script.